2022-01-11T13:48:03.189 INFO:root:teuthology version: 1.1.0-00e09f05 2022-01-11T13:48:03.192 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} --verbose --archive /home/teuthworker/archive/dang-2022-01-11_08:37:46-rgw-wip-dang-testing-distro-basic-smithi/6608060 --owner scheduled_dang@sidious.arb.redhat.com --name dang-2022-01-11_08:37:46-rgw-wip-dang-testing-distro-basic-smithi -- /home/teuthworker/archive/dang-2022-01-11_08:37:46-rgw-wip-dang-testing-distro-basic-smithi/6608060/orig.config.yaml 2022-01-11T13:48:03.261 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-01-11T13:48:03.340 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dang-2022-01-11_08:37:46-rgw-wip-dang-testing-distro-basic-smithi/6608060 branch: wip-dang-testing description: rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} email: null first_in_suite: false job_id: '6608060' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dang-2022-01-11_08:37:46-rgw-wip-dang-testing-distro-basic-smithi nuke-on-error: true os_type: centos os_version: '8.3' overrides: admin_socket: branch: wip-dang-testing ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 17f142526478a11e8f62d3933be552d546c4fd19 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 17f142526478a11e8f62d3933be552d546c4fd19 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 rgw-multisite-tests: args: - tests.py - tests_ps.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-dang-testing sha1: 17f142526478a11e8f62d3933be552d546c4fd19 owner: scheduled_dang@sidious.arb.redhat.com priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 17f142526478a11e8f62d3933be552d546c4fd19 sleep_before_teardown: 0 suite: rgw suite_branch: wip-dang-testing suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_17f142526478a11e8f62d3933be552d546c4fd19/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 17f142526478a11e8f62d3933be552d546c4fd19 targets: smithi161.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+BHf5FlXf0gP7b+a0ejm3R2ApbRE9mcCiTGRp/GWnfGX07cnqSVzw94iCpFRzX6J3cMV8d+NLP9mYK1TQLZTixMkl+OGh+1b6hHxy/MgCPBVK3z8hNjD3cVknHoa3bIs3sYw54hPJV/toQO705SfrzsHeMi4x5339qeAxOR2bnU3KjUI4UJ046tIg5q+DfWZNm7YTOna3Ho0amBKCk5qydCf6LG1ZJP5vYjhGadjW1X9h0r+OhsMaVYf4p6CI7SEDEAy+ZDStL8pwDEes8SAXagWUe39jnXq7hLdg1R9O9Z7Vqf4v2NpTv1sqP6ViqY3fq7rGyYxdZEfw0K/sPufQLlGoW5Vt/Uevi+iu1z5p0pObvUsJwDvQ3QiEzh9bmTQ3BY3nxssCWYRzV4G5iYoUGfUQ3df0bI2tLxQ6Ox38sZUHtFCe5wWkFPh8woIlGhrp9ujJkciUSt+9QtHcfq9O+fw4v0LMBftbSlJK+nYwxGpx8ofcfKk65QS/sFiGDo0= smithi172.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJy9sSSPAjBB6dmli0gWyyA/2N0akOcMO4MO8j5C90Y0c+2I7/6VsrqpueZds/gd9DvL83u49CBfO8h3Zjl5a9HRntGaDGZAqj8DM+3y/1/G2LzgJ2j5WUxIHcdJlRfNWxmGp3iQO0+Dnx0NtQY/JE94GEoLgI+3C8CGaxPnLDQWuNHK+1OMEHUGOppA41xNRzPm7NwEWKEtAh8sa9Fx3VaCXHSqvdxjY1KFg5xSvKkpCF5Wkt60pb8w0GHwM4HqtbbBOwuDIbwQa3cH4ZnriHqypTyVg48xmBOjXjLeClyxete2ExIxSgjlKigQYdU9PKXGO7D+wFunduTokQSA5X2IRH30cy3UvFZyYZpoPHymZCPg+ZJMHic12xNdA36MiSxFOdE/y3ZkQ7Apd+Sz4D6W7+RdBXBPHu67bWf6GCh4WJdCXs1WKAaIimt0SHeNCmYl+jgWy04IgGkm6fd4lGD0VMs+Ub/qj/P8pmyGEzjmizhoBpXpl//Od+07hBfh8= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master teuthology_sha1: 00e09f05e9fb2ab7c7f497f5e446613531743ae2 timestamp: 2022-01-11_08:37:46 tube: smithi user: dang verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.20261 2022-01-11T13:48:03.341 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_17f142526478a11e8f62d3933be552d546c4fd19/qa; will attempt to use it 2022-01-11T13:48:03.341 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_17f142526478a11e8f62d3933be552d546c4fd19/qa/tasks 2022-01-11T13:48:03.342 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-01-11T13:48:03.344 INFO:teuthology.task.internal:Checking packages... 2022-01-11T13:48:03.365 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '17f142526478a11e8f62d3933be552d546c4fd19' 2022-01-11T13:48:03.365 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-01-11T13:48:03.366 INFO:teuthology.packaging:ref: None 2022-01-11T13:48:03.366 INFO:teuthology.packaging:tag: None 2022-01-11T13:48:03.366 INFO:teuthology.packaging:branch: wip-dang-testing 2022-01-11T13:48:03.367 INFO:teuthology.packaging:sha1: 17f142526478a11e8f62d3933be552d546c4fd19 2022-01-11T13:48:03.367 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-dang-testing 2022-01-11T13:48:03.579 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-10004.g17f14252 2022-01-11T13:48:03.580 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-01-11T13:48:03.588 INFO:teuthology.task.internal:no buildpackages task found 2022-01-11T13:48:03.588 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-01-11T13:48:03.609 INFO:teuthology.task.internal:Saving configuration 2022-01-11T13:48:03.634 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-01-11T13:48:03.646 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-11T13:48:03.673 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dang-2022-01-11_08:37:46-rgw-wip-dang-testing-distro-basic-smithi/6608060', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-01-11 13:42:45.128914', 'locked_by': 'scheduled_dang@sidious.arb.redhat.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNJMSLTRQAHHduYFXFem8VmCCPRdnpY7+i7mDGb/MYdwJqnN2OLyZIUY+4h0z16WhQOT+UcdwN1os/KJvAK8IyY='} 2022-01-11T13:48:03.697 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dang-2022-01-11_08:37:46-rgw-wip-dang-testing-distro-basic-smithi/6608060', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-01-11 13:42:45.130271', 'locked_by': 'scheduled_dang@sidious.arb.redhat.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKEY4ZrX7CGncLkzznIqnT0I5hl3h/bUxISxAWpvWsvddtU/VAAVie9qsZFSaLojXZ3w2x6aUsS3fFS3G1UMPGY='} 2022-01-11T13:48:03.698 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-01-11T13:48:03.709 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2022-01-11T13:48:03.709 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2022-01-11T13:48:03.709 INFO:teuthology.run_tasks:Running task console_log... 2022-01-11T13:48:03.795 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff1bd079c80>, signals=[15]) 2022-01-11T13:48:03.796 INFO:teuthology.run_tasks:Running task internal.connect... 2022-01-11T13:48:03.812 INFO:teuthology.task.internal:Opening connections... 2022-01-11T13:48:03.812 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2022-01-11T13:48:03.814 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-11T13:48:03.886 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2022-01-11T13:48:03.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-11T13:48:03.960 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-01-11T13:48:03.972 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2022-01-11T13:48:03.989 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2022-01-11T13:48:03.989 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2022-01-11T13:48:04.043 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Linux" 2022-01-11T13:48:04.044 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="8" 2022-01-11T13:48:04.044 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2022-01-11T13:48:04.044 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2022-01-11T13:48:04.044 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="8" 2022-01-11T13:48:04.044 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el8" 2022-01-11T13:48:04.045 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Linux 8" 2022-01-11T13:48:04.045 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2022-01-11T13:48:04.045 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-11T13:48:04.045 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://centos.org/" 2022-01-11T13:48:04.046 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2022-01-11T13:48:04.046 INFO:teuthology.orchestra.run.smithi161.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2022-01-11T13:48:04.047 INFO:teuthology.orchestra.run.smithi161.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2022-01-11T13:48:04.047 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2022-01-11T13:48:04.066 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2022-01-11T13:48:04.081 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2022-01-11T13:48:04.082 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2022-01-11T13:48:04.135 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Linux" 2022-01-11T13:48:04.136 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="8" 2022-01-11T13:48:04.136 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2022-01-11T13:48:04.136 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2022-01-11T13:48:04.136 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="8" 2022-01-11T13:48:04.137 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el8" 2022-01-11T13:48:04.137 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Linux 8" 2022-01-11T13:48:04.137 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2022-01-11T13:48:04.137 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-11T13:48:04.137 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://centos.org/" 2022-01-11T13:48:04.138 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2022-01-11T13:48:04.138 INFO:teuthology.orchestra.run.smithi172.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2022-01-11T13:48:04.138 INFO:teuthology.orchestra.run.smithi172.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2022-01-11T13:48:04.139 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2022-01-11T13:48:04.157 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-01-11T13:48:04.174 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-01-11T13:48:04.196 INFO:teuthology.task.internal:Checking for old test directory... 2022-01-11T13:48:04.197 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2022-01-11T13:48:04.200 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2022-01-11T13:48:04.214 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-01-11T13:48:04.231 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-01-11T13:48:04.232 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2022-01-11T13:48:04.254 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2022-01-11T13:48:04.276 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-01-11T13:48:04.334 INFO:teuthology.run_tasks:Running task kernel... 2022-01-11T13:48:04.350 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-01-11T13:48:04.351 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-01-11T13:48:04.351 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2022-01-11T13:48:04.352 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2022-01-11T13:48:04.366 INFO:teuthology.orchestra.run.smithi172.stdout:4.18.0-305.10.2.el8_4.x86_64 2022-01-11T13:48:04.367 INFO:teuthology.task.kernel:Running kernel on smithi172: 4.18.0-305.10.2.el8_4.x86_64 2022-01-11T13:48:04.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2022-01-11T13:48:04.368 INFO:teuthology.orchestra.run.smithi161.stdout:4.18.0-305.10.2.el8_4.x86_64 2022-01-11T13:48:04.369 INFO:teuthology.task.kernel:Running kernel on smithi161: 4.18.0-305.10.2.el8_4.x86_64 2022-01-11T13:48:04.369 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum install -y kernel 2022-01-11T13:48:06.577 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-8 - AppStream 34 MB/s | 8.4 MB 00:00 2022-01-11T13:48:06.632 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-8 - AppStream 35 MB/s | 8.4 MB 00:00 2022-01-11T13:48:08.003 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-8 - Base 41 MB/s | 4.6 MB 00:00 2022-01-11T13:48:08.064 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-8 - Base 39 MB/s | 4.6 MB 00:00 2022-01-11T13:48:08.722 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-8 - Extras 162 kB/s | 10 kB 00:00 2022-01-11T13:48:08.774 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-8 - Extras 161 kB/s | 10 kB 00:00 2022-01-11T13:48:10.410 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Linux 8 - AppStream 5.2 MB/s | 8.4 MB 00:01 2022-01-11T13:48:11.431 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Linux 8 - AppStream 3.2 MB/s | 8.4 MB 00:02 2022-01-11T13:48:12.131 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Linux 8 - BaseOS 13 MB/s | 4.6 MB 00:00 2022-01-11T13:48:13.004 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Linux 8 - Extras 46 kB/s | 10 kB 00:00 2022-01-11T13:48:13.158 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-8 - PowerTools 25 MB/s | 2.3 MB 00:00 2022-01-11T13:48:13.317 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Linux 8 - BaseOS 9.4 MB/s | 4.6 MB 00:00 2022-01-11T13:48:14.275 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Linux 8 - Extras 35 kB/s | 10 kB 00:00 2022-01-11T13:48:14.422 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-8 - PowerTools 26 MB/s | 2.3 MB 00:00 2022-01-11T13:48:15.712 INFO:teuthology.orchestra.run.smithi172.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2022-01-11T13:48:16.565 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 14 MB/s | 11 MB 00:00 2022-01-11T13:48:16.733 INFO:teuthology.orchestra.run.smithi161.stdout:Copr repo for python3-asyncssh owned by ceph 1.8 kB/s | 3.5 kB 00:01 2022-01-11T13:48:17.587 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 14 MB/s | 11 MB 00:00 2022-01-11T13:48:18.987 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2022-01-11T13:48:19.985 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 4.8 MB/s | 24 kB 00:00 2022-01-11T13:48:20.969 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-4.18.0-305.10.2.el8_4.x86_64 is already installed. 2022-01-11T13:48:20.969 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2022-01-11T13:48:21.242 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:48:21.243 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:48:21.243 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2022-01-11T13:48:21.243 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:48:21.244 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:48:21.244 INFO:teuthology.orchestra.run.smithi172.stdout: kernel x86_64 4.18.0-348.7.1.el8_5 CentOS-Base 7.0 M 2022-01-11T13:48:21.244 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2022-01-11T13:48:21.244 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core x86_64 4.18.0-348.7.1.el8_5 CentOS-Base 38 M 2022-01-11T13:48:21.245 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules x86_64 4.18.0-348.7.1.el8_5 CentOS-Base 30 M 2022-01-11T13:48:21.245 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:48:21.245 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:48:21.245 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:48:21.246 INFO:teuthology.orchestra.run.smithi172.stdout:Install 3 Packages 2022-01-11T13:48:21.246 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:48:21.247 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 74 M 2022-01-11T13:48:21.247 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 90 M 2022-01-11T13:48:21.247 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:48:21.475 INFO:teuthology.orchestra.run.smithi172.stdout:(1/3): kernel-4.18.0-348.7.1.el8_5.x86_64.rpm 30 MB/s | 7.0 MB 00:00 2022-01-11T13:48:21.929 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-4.18.0-305.10.2.el8_4.x86_64 is already installed. 2022-01-11T13:48:21.930 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2022-01-11T13:48:22.065 INFO:teuthology.orchestra.run.smithi172.stdout:(2/3): kernel-core-4.18.0-348.7.1.el8_5.x86_64. 46 MB/s | 38 MB 00:00 2022-01-11T13:48:22.240 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:48:22.241 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:48:22.241 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2022-01-11T13:48:22.241 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:48:22.241 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:48:22.242 INFO:teuthology.orchestra.run.smithi161.stdout: kernel x86_64 4.18.0-348.7.1.el8_5 CentOS-Base 7.0 M 2022-01-11T13:48:22.242 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2022-01-11T13:48:22.242 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-core x86_64 4.18.0-348.7.1.el8_5 CentOS-Base 38 M 2022-01-11T13:48:22.242 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules x86_64 4.18.0-348.7.1.el8_5 CentOS-Base 30 M 2022-01-11T13:48:22.243 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:48:22.243 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:48:22.243 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:48:22.243 INFO:teuthology.orchestra.run.smithi161.stdout:Install 3 Packages 2022-01-11T13:48:22.244 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:48:22.244 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 74 M 2022-01-11T13:48:22.244 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 90 M 2022-01-11T13:48:22.244 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:48:22.416 INFO:teuthology.orchestra.run.smithi172.stdout:(3/3): kernel-modules-4.18.0-348.7.1.el8_5.x86_ 25 MB/s | 30 MB 00:01 2022-01-11T13:48:22.420 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:48:22.421 INFO:teuthology.orchestra.run.smithi172.stdout:Total 63 MB/s | 74 MB 00:01 2022-01-11T13:48:22.451 INFO:teuthology.orchestra.run.smithi161.stdout:(1/3): kernel-4.18.0-348.7.1.el8_5.x86_64.rpm 33 MB/s | 7.0 MB 00:00 2022-01-11T13:48:22.552 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:48:22.896 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:48:22.897 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:48:22.927 INFO:teuthology.orchestra.run.smithi161.stdout:(2/3): kernel-modules-4.18.0-348.7.1.el8_5.x86_ 43 MB/s | 30 MB 00:00 2022-01-11T13:48:23.453 INFO:teuthology.orchestra.run.smithi161.stdout:(3/3): kernel-core-4.18.0-348.7.1.el8_5.x86_64. 31 MB/s | 38 MB 00:01 2022-01-11T13:48:23.457 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:48:23.458 INFO:teuthology.orchestra.run.smithi161.stdout:Total 61 MB/s | 74 MB 00:01 2022-01-11T13:48:23.591 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:48:23.921 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:48:23.921 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:48:24.120 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:48:24.133 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:48:25.171 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:48:25.182 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:48:26.669 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:48:27.146 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-core-4.18.0-348.7.1.el8_5.x86_64 1/3 2022-01-11T13:48:27.678 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:48:28.133 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-core-4.18.0-348.7.1.el8_5.x86_64 1/3 2022-01-11T13:48:30.402 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-4.18.0-348.7.1.el8_5.x86_64 1/3 2022-01-11T13:48:30.770 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-4.18.0-348.7.1.el8_5.x86_64 2/3 2022-01-11T13:48:31.345 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-core-4.18.0-348.7.1.el8_5.x86_64 1/3 2022-01-11T13:48:31.700 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-modules-4.18.0-348.7.1.el8_5.x86_64 2/3 2022-01-11T13:48:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-4.18.0-348.7.1.el8_5.x86_64 2/3 2022-01-11T13:48:34.047 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-4.18.0-348.7.1.el8_5.x86_64 3/3 2022-01-11T13:48:34.762 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-4.18.0-348.7.1.el8_5.x86_64 2/3 2022-01-11T13:48:35.001 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-4.18.0-348.7.1.el8_5.x86_64 3/3 2022-01-11T13:48:54.653 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-4.18.0-348.7.1.el8_5.x86_64 3/3 2022-01-11T13:48:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-4.18.0-348.7.1.el8_5.x86_64 3/3 2022-01-11T13:48:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-4.18.0-348.7.1.el8_5.x86_64 1/3 2022-01-11T13:48:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-core-4.18.0-348.7.1.el8_5.x86_64 2/3 2022-01-11T13:48:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-4.18.0-348.7.1.el8_5.x86_64 3/3 2022-01-11T13:48:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:48:55.380 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:48:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core-4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:48:55.381 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:48:55.450 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel --last | head -n 1 2022-01-11T13:48:55.607 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-4.18.0-348.7.1.el8_5.x86_64 Tue 11 Jan 2022 01:48:33 PM UTC 2022-01-11T13:48:55.609 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:55.609 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.10.2.el8_4.x86_64 Expected: 4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:55.609 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-01-11T13:48:55.610 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-01-11T13:48:55.610 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2022-01-11T13:48:55.610 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2022-01-11T13:48:55.610 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel --last | head -n 1 2022-01-11T13:48:55.687 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-core-4.18.0-348.7.1.el8_5.x86_64 3/3 2022-01-11T13:48:55.750 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-4.18.0-348.7.1.el8_5.x86_64 Tue 11 Jan 2022 01:48:33 PM UTC 2022-01-11T13:48:55.752 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:55.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo rpm -qi grub2-tools 2022-01-11T13:48:55.783 INFO:teuthology.orchestra.run.smithi172.stdout:Name : grub2-tools 2022-01-11T13:48:55.783 INFO:teuthology.orchestra.run.smithi172.stdout:Epoch : 1 2022-01-11T13:48:55.783 INFO:teuthology.orchestra.run.smithi172.stdout:Version : 2.02 2022-01-11T13:48:55.784 INFO:teuthology.orchestra.run.smithi172.stdout:Release : 90.el8 2022-01-11T13:48:55.784 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: x86_64 2022-01-11T13:48:55.784 INFO:teuthology.orchestra.run.smithi172.stdout:Install Date: Wed 11 Aug 2021 06:38:04 PM UTC 2022-01-11T13:48:55.784 INFO:teuthology.orchestra.run.smithi172.stdout:Group : System Environment/Base 2022-01-11T13:48:55.785 INFO:teuthology.orchestra.run.smithi172.stdout:Size : 9197207 2022-01-11T13:48:55.785 INFO:teuthology.orchestra.run.smithi172.stdout:License : GPLv3+ 2022-01-11T13:48:55.785 INFO:teuthology.orchestra.run.smithi172.stdout:Signature : RSA/SHA256, Tue 08 Sep 2020 07:14:39 PM UTC, Key ID 05b555b38483c65d 2022-01-11T13:48:55.785 INFO:teuthology.orchestra.run.smithi172.stdout:Source RPM : grub2-2.02-90.el8.src.rpm 2022-01-11T13:48:55.786 INFO:teuthology.orchestra.run.smithi172.stdout:Build Date : Tue 08 Sep 2020 07:00:29 PM UTC 2022-01-11T13:48:55.786 INFO:teuthology.orchestra.run.smithi172.stdout:Build Host : kbuilder.bsys.centos.org 2022-01-11T13:48:55.786 INFO:teuthology.orchestra.run.smithi172.stdout:Relocations : (not relocatable) 2022-01-11T13:48:55.786 INFO:teuthology.orchestra.run.smithi172.stdout:Packager : CentOS BuildSystem 2022-01-11T13:48:55.786 INFO:teuthology.orchestra.run.smithi172.stdout:Vendor : CentOS 2022-01-11T13:48:55.787 INFO:teuthology.orchestra.run.smithi172.stdout:URL : http://www.gnu.org/software/grub/ 2022-01-11T13:48:55.787 INFO:teuthology.orchestra.run.smithi172.stdout:Summary : Support tools for GRUB. 2022-01-11T13:48:55.787 INFO:teuthology.orchestra.run.smithi172.stdout:Description : 2022-01-11T13:48:55.787 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:48:55.788 INFO:teuthology.orchestra.run.smithi172.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2022-01-11T13:48:55.788 INFO:teuthology.orchestra.run.smithi172.stdout:customizable bootloader with modular architecture. It supports a rich 2022-01-11T13:48:55.788 INFO:teuthology.orchestra.run.smithi172.stdout:variety of kernel formats, file systems, computer architectures and 2022-01-11T13:48:55.788 INFO:teuthology.orchestra.run.smithi172.stdout:hardware devices. 2022-01-11T13:48:55.789 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:48:55.789 INFO:teuthology.orchestra.run.smithi172.stdout:This subpackage provides tools for support of all platforms. 2022-01-11T13:48:55.790 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2022-01-11T13:48:55.790 INFO:teuthology.task.kernel:Updating grub on smithi172 to boot 4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:55.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2022-01-11T13:48:55.997 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-4.18.0-348.7.1.el8_5.x86_64 3/3 2022-01-11T13:48:55.998 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-4.18.0-348.7.1.el8_5.x86_64 1/3 2022-01-11T13:48:55.998 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-core-4.18.0-348.7.1.el8_5.x86_64 2/3 2022-01-11T13:48:56.388 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-modules-4.18.0-348.7.1.el8_5.x86_64 3/3 2022-01-11T13:48:56.388 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:48:56.389 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:48:56.389 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:56.389 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-core-4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:56.389 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:56.390 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:48:56.390 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:48:56.457 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel --last | head -n 1 2022-01-11T13:48:56.609 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-4.18.0-348.7.1.el8_5.x86_64 Tue 11 Jan 2022 01:48:34 PM UTC 2022-01-11T13:48:56.611 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:56.611 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.10.2.el8_4.x86_64 Expected: 4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:56.611 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-01-11T13:48:56.612 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-01-11T13:48:56.612 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2022-01-11T13:48:56.612 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi161.front.sepia.ceph.com, path=None, version=distro) 2022-01-11T13:48:56.613 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel --last | head -n 1 2022-01-11T13:48:56.752 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-4.18.0-348.7.1.el8_5.x86_64 Tue 11 Jan 2022 01:48:34 PM UTC 2022-01-11T13:48:56.754 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:56.754 DEBUG:teuthology.orchestra.run.smithi161:> sudo rpm -qi grub2-tools 2022-01-11T13:48:56.785 INFO:teuthology.orchestra.run.smithi161.stdout:Name : grub2-tools 2022-01-11T13:48:56.785 INFO:teuthology.orchestra.run.smithi161.stdout:Epoch : 1 2022-01-11T13:48:56.785 INFO:teuthology.orchestra.run.smithi161.stdout:Version : 2.02 2022-01-11T13:48:56.786 INFO:teuthology.orchestra.run.smithi161.stdout:Release : 90.el8 2022-01-11T13:48:56.786 INFO:teuthology.orchestra.run.smithi161.stdout:Architecture: x86_64 2022-01-11T13:48:56.786 INFO:teuthology.orchestra.run.smithi161.stdout:Install Date: Wed 11 Aug 2021 06:38:04 PM UTC 2022-01-11T13:48:56.786 INFO:teuthology.orchestra.run.smithi161.stdout:Group : System Environment/Base 2022-01-11T13:48:56.787 INFO:teuthology.orchestra.run.smithi161.stdout:Size : 9197207 2022-01-11T13:48:56.787 INFO:teuthology.orchestra.run.smithi161.stdout:License : GPLv3+ 2022-01-11T13:48:56.787 INFO:teuthology.orchestra.run.smithi161.stdout:Signature : RSA/SHA256, Tue 08 Sep 2020 07:14:39 PM UTC, Key ID 05b555b38483c65d 2022-01-11T13:48:56.787 INFO:teuthology.orchestra.run.smithi161.stdout:Source RPM : grub2-2.02-90.el8.src.rpm 2022-01-11T13:48:56.788 INFO:teuthology.orchestra.run.smithi161.stdout:Build Date : Tue 08 Sep 2020 07:00:29 PM UTC 2022-01-11T13:48:56.788 INFO:teuthology.orchestra.run.smithi161.stdout:Build Host : kbuilder.bsys.centos.org 2022-01-11T13:48:56.788 INFO:teuthology.orchestra.run.smithi161.stdout:Relocations : (not relocatable) 2022-01-11T13:48:56.788 INFO:teuthology.orchestra.run.smithi161.stdout:Packager : CentOS BuildSystem 2022-01-11T13:48:56.788 INFO:teuthology.orchestra.run.smithi161.stdout:Vendor : CentOS 2022-01-11T13:48:56.789 INFO:teuthology.orchestra.run.smithi161.stdout:URL : http://www.gnu.org/software/grub/ 2022-01-11T13:48:56.789 INFO:teuthology.orchestra.run.smithi161.stdout:Summary : Support tools for GRUB. 2022-01-11T13:48:56.789 INFO:teuthology.orchestra.run.smithi161.stdout:Description : 2022-01-11T13:48:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:48:56.790 INFO:teuthology.orchestra.run.smithi161.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2022-01-11T13:48:56.790 INFO:teuthology.orchestra.run.smithi161.stdout:customizable bootloader with modular architecture. It supports a rich 2022-01-11T13:48:56.790 INFO:teuthology.orchestra.run.smithi161.stdout:variety of kernel formats, file systems, computer architectures and 2022-01-11T13:48:56.790 INFO:teuthology.orchestra.run.smithi161.stdout:hardware devices. 2022-01-11T13:48:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:48:56.791 INFO:teuthology.orchestra.run.smithi161.stdout:This subpackage provides tools for support of all platforms. 2022-01-11T13:48:56.792 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2022-01-11T13:48:56.792 INFO:teuthology.task.kernel:Updating grub on smithi161 to boot 4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:56.793 DEBUG:teuthology.orchestra.run.smithi161:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2022-01-11T13:48:57.309 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2022-01-11T13:48:58.384 INFO:teuthology.orchestra.run.smithi161.stderr:Generating grub configuration file ... 2022-01-11T13:48:59.424 INFO:teuthology.orchestra.run.smithi172.stderr:done 2022-01-11T13:48:59.426 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2022-01-11T13:48:59.464 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.ealsVYfGv1 2022-01-11T13:48:59.465 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ealsVYfGv1 2022-01-11T13:48:59.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.ealsVYfGv1 2022-01-11T13:48:59.684 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.ealsVYfGv1 is 6KB 2022-01-11T13:48:59.733 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.ealsVYfGv1 2022-01-11T13:48:59.747 DEBUG:teuthology.orchestra.run.smithi172:> sudo /bin/ls /boot/loader/entries || true 2022-01-11T13:48:59.810 INFO:teuthology.orchestra.run.smithi172.stdout:ecc66dffcd264e5181894e69c53ff13c-0-rescue.conf 2022-01-11T13:48:59.811 INFO:teuthology.orchestra.run.smithi172.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-240.el8.x86_64.conf 2022-01-11T13:48:59.811 INFO:teuthology.orchestra.run.smithi172.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.10.2.el8_4.x86_64.conf 2022-01-11T13:48:59.811 INFO:teuthology.orchestra.run.smithi172.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-348.7.1.el8_5.x86_64.conf 2022-01-11T13:48:59.813 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-set-default ecc66dffcd264e5181894e69c53ff13c-4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:48:59.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2022-01-11T13:49:00.559 INFO:teuthology.orchestra.run.smithi161.stderr:done 2022-01-11T13:49:00.560 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2022-01-11T13:49:00.606 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.R13yvNro5L 2022-01-11T13:49:00.607 DEBUG:teuthology.orchestra.run.smithi161:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.R13yvNro5L 2022-01-11T13:49:00.647 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.R13yvNro5L 2022-01-11T13:49:00.817 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.R13yvNro5L is 6KB 2022-01-11T13:49:00.865 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.R13yvNro5L 2022-01-11T13:49:00.879 DEBUG:teuthology.orchestra.run.smithi161:> sudo /bin/ls /boot/loader/entries || true 2022-01-11T13:49:00.941 INFO:teuthology.orchestra.run.smithi161.stdout:ecc66dffcd264e5181894e69c53ff13c-0-rescue.conf 2022-01-11T13:49:00.941 INFO:teuthology.orchestra.run.smithi161.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-240.el8.x86_64.conf 2022-01-11T13:49:00.941 INFO:teuthology.orchestra.run.smithi161.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.10.2.el8_4.x86_64.conf 2022-01-11T13:49:00.942 INFO:teuthology.orchestra.run.smithi161.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-348.7.1.el8_5.x86_64.conf 2022-01-11T13:49:00.943 DEBUG:teuthology.orchestra.run.smithi161:> sudo grub2-set-default ecc66dffcd264e5181894e69c53ff13c-4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:49:01.089 DEBUG:teuthology.orchestra.run.smithi161:> sudo shutdown -r now 2022-01-11T13:49:29.970 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2022-01-11T13:49:29.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2022-01-11T13:49:29.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-11T13:49:31.094 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2022-01-11T13:49:31.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2022-01-11T13:49:31.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-11T13:51:04.544 DEBUG:teuthology.orchestra.remote:timed out 2022-01-11T13:51:05.773 DEBUG:teuthology.orchestra.remote:timed out 2022-01-11T13:51:34.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2022-01-11T13:51:34.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-11T13:51:34.878 DEBUG:teuthology.orchestra.run.smithi172:> true 2022-01-11T13:51:35.185 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2022-01-11T13:51:35.185 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.18.0-348.7.1.el8_5.x86_64"... 2022-01-11T13:51:35.186 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2022-01-11T13:51:35.201 INFO:teuthology.orchestra.run.smithi172.stdout:4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:51:35.201 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.7.1.el8_5.x86_64 vs 4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:51:35.201 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-01-11T13:51:35.201 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2022-01-11T13:51:35.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2022-01-11T13:51:35.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-11T13:51:36.035 DEBUG:teuthology.orchestra.run.smithi161:> true 2022-01-11T13:51:36.202 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2022-01-11T13:51:36.203 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-01-11T13:51:36.271 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2022-01-11T13:51:36.284 DEBUG:teuthology.parallel:result is None 2022-01-11T13:51:36.334 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2022-01-11T13:51:36.335 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.18.0-348.7.1.el8_5.x86_64"... 2022-01-11T13:51:36.335 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2022-01-11T13:51:36.350 INFO:teuthology.orchestra.run.smithi161.stdout:4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:51:36.351 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.7.1.el8_5.x86_64 vs 4.18.0-348.7.1.el8_5.x86_64 2022-01-11T13:51:36.351 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-01-11T13:51:36.352 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2022-01-11T13:51:37.353 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2022-01-11T13:51:37.354 DEBUG:teuthology.orchestra.run.smithi161:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-01-11T13:51:37.442 INFO:teuthology.orchestra.run.smithi161.stdout:ttyS1 2022-01-11T13:51:37.454 DEBUG:teuthology.parallel:result is None 2022-01-11T13:51:37.454 INFO:teuthology.run_tasks:Running task internal.base... 2022-01-11T13:51:37.466 INFO:teuthology.task.internal:Creating test directory... 2022-01-11T13:51:37.467 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-01-11T13:51:37.469 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-01-11T13:51:37.487 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-01-11T13:51:37.562 INFO:teuthology.run_tasks:Running task internal.archive... 2022-01-11T13:51:37.593 INFO:teuthology.task.internal:Creating archive directory... 2022-01-11T13:51:37.594 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-01-11T13:51:37.597 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-01-11T13:51:37.628 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-01-11T13:51:37.639 INFO:teuthology.task.internal:Enabling coredump saving... 2022-01-11T13:51:37.640 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-01-11T13:51:37.662 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-01-11T13:51:37.689 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-01-11T13:51:37.700 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-01-11T13:51:37.702 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-01-11T13:51:37.714 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-01-11T13:51:37.715 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-01-11T13:51:37.727 INFO:teuthology.task.internal:Configuring sudo... 2022-01-11T13:51:37.728 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-01-11T13:51:37.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-01-11T13:51:37.782 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-01-11T13:51:37.793 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-01-11T13:51:37.794 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-01-11T13:51:37.812 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-01-11T13:51:37.838 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-01-11T13:51:37.910 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-01-11T13:51:37.968 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-01-11T13:51:38.049 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-01-11T13:51:38.091 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:51:38.092 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-01-11T13:51:38.157 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-01-11T13:51:38.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-01-11T13:51:38.265 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-01-11T13:51:38.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-01-11T13:51:38.349 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T13:51:38.350 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-01-11T13:51:38.415 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2022-01-11T13:51:38.418 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2022-01-11T13:51:38.470 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-01-11T13:51:38.502 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-01-11T13:51:38.864 INFO:teuthology.run_tasks:Running task internal.timer... 2022-01-11T13:51:38.874 INFO:teuthology.task.internal:Starting timer... 2022-01-11T13:51:38.874 INFO:teuthology.run_tasks:Running task pcp... 2022-01-11T13:51:38.912 INFO:teuthology.run_tasks:Running task selinux... 2022-01-11T13:51:38.944 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-01-11T13:51:38.945 DEBUG:teuthology.orchestra.run.smithi161:> sudo service auditd rotate 2022-01-11T13:51:39.015 INFO:teuthology.orchestra.run.smithi161.stdout:Rotating logs: [ OK ] 2022-01-11T13:51:39.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2022-01-11T13:51:39.121 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: [ OK ] 2022-01-11T13:51:39.123 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-01-11T13:51:39.123 DEBUG:teuthology.orchestra.run.smithi161:> /usr/sbin/getenforce 2022-01-11T13:51:39.147 INFO:teuthology.orchestra.run.smithi161.stdout:Permissive 2022-01-11T13:51:39.147 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2022-01-11T13:51:39.163 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2022-01-11T13:51:39.164 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi161.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2022-01-11T13:51:39.164 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-01-11T13:51:39.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-11T13:51:39.213 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-01-11T13:51:39.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-11T13:51:39.240 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-01-11T13:51:39.240 DEBUG:teuthology.orchestra.run.smithi161:> sudo /usr/sbin/setenforce permissive 2022-01-11T13:51:39.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2022-01-11T13:51:39.314 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-01-11T13:51:39.325 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-01-11T13:51:39.325 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-01-11T13:51:39.345 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-01-11T13:51:39.347 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi161.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-01-11T13:51:39.716 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2022-01-11T13:51:40.533 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2022-01-11T13:51:43.456 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2022-01-11T13:51:43.566 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:43 +0000 (0:00:00.126) 0:00:00.126 ******* 2022-01-11T13:51:43.577 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:43 +0000 (0:00:00.011) 0:00:00.137 ******* 2022-01-11T13:51:45.032 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi172.front.sepia.ceph.com] 2022-01-11T13:51:45.080 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:45 +0000 (0:00:01.502) 0:00:01.640 ******* 2022-01-11T13:51:45.344 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2022-01-11T13:51:45.346 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2022-01-11T13:51:45.371 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:45 +0000 (0:00:00.291) 0:00:01.931 ******* 2022-01-11T13:51:45.896 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2022-01-11T13:51:45.897 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2022-01-11T13:51:45.929 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:45 +0000 (0:00:00.558) 0:00:02.489 ******* 2022-01-11T13:51:45.991 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2022-01-11T13:51:45.993 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:51:46.013 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:46 +0000 (0:00:00.083) 0:00:02.573 ******* 2022-01-11T13:51:46.037 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi161.front.sepia.ceph.com] 2022-01-11T13:51:46.083 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:46 +0000 (0:00:00.070) 0:00:02.643 ******* 2022-01-11T13:51:46.267 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2022-01-11T13:51:46.269 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2022-01-11T13:51:46.290 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:46 +0000 (0:00:00.206) 0:00:02.850 ******* 2022-01-11T13:51:46.297 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2022-01-11T13:51:46.299 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2022-01-11T13:51:46.320 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:46 +0000 (0:00:00.030) 0:00:02.880 ******* 2022-01-11T13:51:46.607 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2022-01-11T13:51:46.608 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2022-01-11T13:51:46.629 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:46 +0000 (0:00:00.308) 0:00:03.189 ******* 2022-01-11T13:51:46.697 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2022-01-11T13:51:46.698 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:51:46.719 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:46 +0000 (0:00:00.089) 0:00:03.279 ******* 2022-01-11T13:51:46.900 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2022-01-11T13:51:46.902 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2022-01-11T13:51:46.923 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:46 +0000 (0:00:00.203) 0:00:03.483 ******* 2022-01-11T13:51:47.191 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2022-01-11T13:51:47.193 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1641909106.338416-26946-166300323197471/source", "state": "file", "uid": 0} 2022-01-11T13:51:47.215 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:47 +0000 (0:00:00.292) 0:00:03.775 ******* 2022-01-11T13:51:47.527 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1641909106.9687262-27011-61869176375797/source", "state": "file", "uid": 0} 2022-01-11T13:51:47.550 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:47 +0000 (0:00:00.334) 0:00:04.110 ******* 2022-01-11T13:51:47.770 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtC 2022-01-11T13:51:47.772 INFO:teuthology.task.ansible.out:toHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2022-01-11T13:51:47.866 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2022-01-11T13:51:47.868 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2022-01-11T13:51:47.932 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2022-01-11T13:51:47.934 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2022-01-11T13:51:47.947 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Tuesday 11 January 2022 13:51:47 +0000 (0:00:00.397) 0:00:04.507 ******* 2022-01-11T13:51:48.468 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] 2022-01-11T13:51:48.486 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] 2022-01-11T13:51:48.502 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** 2022-01-11T13:51:48.504 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:48 +0000 (0:00:00.555) 0:00:05.063 ******* 2022-01-11T13:51:48.527 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2022-01-11T13:51:48.538 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2022-01-11T13:51:48.543 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2022-01-11T13:51:48.697 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:48 +0000 (0:00:00.194) 0:00:05.257 ******* 2022-01-11T13:51:48.718 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:48 +0000 (0:00:00.020) 0:00:05.278 ******* 2022-01-11T13:51:49.331 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi161.front.sepia.ceph.com] 2022-01-11T13:51:49.383 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2022-01-11T13:51:49.385 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:49 +0000 (0:00:00.665) 0:00:05.943 ******* 2022-01-11T13:51:49.400 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] 2022-01-11T13:51:49.452 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:49 +0000 (0:00:00.068) 0:00:06.012 ******* 2022-01-11T13:51:50.056 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2022-01-11T13:51:50.058 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.188639", "end": "2022-01-11 13:51:50.032449", "rc": 0, "start": "2022-01-11 13:51:49.843810", "stderr": "", "stderr_lines": [], "stdout": "b39424e053d42ee607f17b88c1886c2d02811d8f", "stdout_lines": ["b39424e053d42ee607f17b88c1886c2d02811d8f"]} 2022-01-11T13:51:50.078 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:50 +0000 (0:00:00.626) 0:00:06.639 ******* 2022-01-11T13:51:50.096 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.214226", "end": "2022-01-11 13:51:50.069118", "rc": 0, "start": "2022-01-11 13:51:49.854892", "stderr": "", "stderr_lines": [], "stdout": "b39424e053d42ee607f17b88c1886c2d02811d8f", "stdout_lines": ["b39424e053d42ee607f17b88c1886c2d02811d8f"]} 2022-01-11T13:51:50.120 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:50 +0000 (0:00:00.041) 0:00:06.680 ******* 2022-01-11T13:51:50.273 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2022-01-11T13:51:50.275 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.011870", "end": "2022-01-11 13:51:50.258079", "failed_when_result": false, "rc": 0, "start": "2022-01-11 13:51:50.246209", "stderr": "", "stderr_lines": [], "stdout": "c29274177b79f72740fc714fa28b55400cd0737f", "stdout_lines": ["c29274177b79f72740fc714fa28b55400cd0737f"]} 2022-01-11T13:51:50.298 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:50 +0000 (0:00:00.177) 0:00:06.858 ******* 2022-01-11T13:51:50.326 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013319", "end": "2022-01-11 13:51:50.303630", "failed_when_result": false, "rc": 0, "start": "2022-01-11 13:51:50.290311", "stderr": "", "stderr_lines": [], "stdout": "c29274177b79f72740fc714fa28b55400cd0737f", "stdout_lines": ["c29274177b79f72740fc714fa28b55400cd0737f"]} 2022-01-11T13:51:50.350 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:50 +0000 (0:00:00.051) 0:00:06.909 ******* 2022-01-11T13:51:50.366 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2022-01-11T13:51:50.368 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:51:50.391 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:50 +0000 (0:00:00.041) 0:00:06.951 ******* 2022-01-11T13:51:50.419 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:51:50.442 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:50 +0000 (0:00:00.051) 0:00:07.003 ******* 2022-01-11T13:51:50.470 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2022-01-11T13:51:50.472 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:51:50.494 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:50 +0000 (0:00:00.051) 0:00:07.054 ******* 2022-01-11T13:51:50.521 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:51:50.545 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:50 +0000 (0:00:00.051) 0:00:07.105 ******* 2022-01-11T13:51:50.594 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2022-01-11T13:51:50.595 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:51:50.617 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:50 +0000 (0:00:00.071) 0:00:07.177 ******* 2022-01-11T13:51:50.633 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:51:50.656 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:50 +0000 (0:00:00.039) 0:00:07.216 ******* 2022-01-11T13:51:50.683 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2022-01-11T13:51:50.685 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:51:50.708 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:50 +0000 (0:00:00.051) 0:00:07.268 ******* 2022-01-11T13:51:50.737 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:51:50.760 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:50 +0000 (0:00:00.051) 0:00:07.320 ******* 2022-01-11T13:51:50.776 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2022-01-11T13:51:50.778 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:51:50.800 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:50 +0000 (0:00:00.040) 0:00:07.361 ******* 2022-01-11T13:51:50.839 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:51:50.865 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:50 +0000 (0:00:00.064) 0:00:07.425 ******* 2022-01-11T13:51:51.041 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2022-01-11T13:51:51.043 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2022-01-11T13:51:51.066 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:51 +0000 (0:00:00.200) 0:00:07.626 ******* 2022-01-11T13:51:51.095 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2022-01-11T13:51:51.119 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:51:51 +0000 (0:00:00.053) 0:00:07.679 ******* 2022-01-11T13:51:51.346 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2022-01-11T13:51:51.348 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2022-01-11T13:51:51.411 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2022-01-11T13:51:51.537 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2022-01-11T13:51:51.611 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2022-01-11T13:51:51.740 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2022-01-11T13:51:51.813 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2022-01-11T13:51:51.939 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2022-01-11T13:51:52.012 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2022-01-11T13:51:52.127 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2022-01-11T13:51:52.201 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2022-01-11T13:51:52.327 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2022-01-11T13:51:52.391 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2022-01-11T13:51:52.516 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2022-01-11T13:51:52.590 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2022-01-11T13:51:52.716 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2022-01-11T13:51:52.791 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2022-01-11T13:51:52.919 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2022-01-11T13:51:52.983 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2022-01-11T13:51:53.111 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2022-01-11T13:51:53.186 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2022-01-11T13:51:53.303 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2022-01-11T13:51:53.379 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2022-01-11T13:51:53.498 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2022-01-11T13:51:53.582 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2022-01-11T13:51:53.699 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2022-01-11T13:51:53.775 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2022-01-11T13:51:53.902 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2022-01-11T13:51:53.977 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2022-01-11T13:51:54.093 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2022-01-11T13:51:54.178 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2022-01-11T13:51:54.295 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2022-01-11T13:51:54.371 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2022-01-11T13:51:54.488 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2022-01-11T13:51:54.573 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2022-01-11T13:51:54.698 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2022-01-11T13:51:54.784 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2022-01-11T13:51:54.890 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2022-01-11T13:51:54.975 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2022-01-11T13:51:55.080 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2022-01-11T13:51:55.175 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2022-01-11T13:51:55.279 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2022-01-11T13:51:55.374 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2022-01-11T13:51:55.474 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2022-01-11T13:51:55.568 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2022-01-11T13:51:55.673 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2022-01-11T13:51:55.766 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2022-01-11T13:51:55.871 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2022-01-11T13:51:55.965 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2022-01-11T13:51:56.069 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2022-01-11T13:51:56.165 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2022-01-11T13:51:56.260 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2022-01-11T13:51:56.355 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2022-01-11T13:51:56.460 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2022-01-11T13:51:56.555 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2022-01-11T13:51:56.660 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2022-01-11T13:51:56.745 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2022-01-11T13:51:56.849 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2022-01-11T13:51:56.954 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2022-01-11T13:51:57.049 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2022-01-11T13:51:57.145 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2022-01-11T13:51:57.239 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2022-01-11T13:51:57.334 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2022-01-11T13:51:57.438 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2022-01-11T13:51:57.533 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2022-01-11T13:51:57.639 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2022-01-11T13:51:57.733 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2022-01-11T13:51:57.838 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2022-01-11T13:51:57.935 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2022-01-11T13:51:58.043 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2022-01-11T13:51:58.137 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2022-01-11T13:51:58.231 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2022-01-11T13:51:58.327 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2022-01-11T13:51:58.433 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2022-01-11T13:51:58.528 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2022-01-11T13:51:58.632 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2022-01-11T13:51:58.727 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2022-01-11T13:51:58.822 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2022-01-11T13:51:58.916 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2022-01-11T13:51:59.021 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2022-01-11T13:51:59.116 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2022-01-11T13:51:59.210 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2022-01-11T13:51:59.314 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2022-01-11T13:51:59.409 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2022-01-11T13:51:59.513 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2022-01-11T13:51:59.608 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2022-01-11T13:51:59.703 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2022-01-11T13:51:59.798 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2022-01-11T13:51:59.903 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2022-01-11T13:51:59.999 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2022-01-11T13:52:00.105 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2022-01-11T13:52:00.190 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2022-01-11T13:52:00.296 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2022-01-11T13:52:00.390 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2022-01-11T13:52:00.497 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2022-01-11T13:52:00.593 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2022-01-11T13:52:00.700 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2022-01-11T13:52:00.784 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2022-01-11T13:52:00.900 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2022-01-11T13:52:00.985 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2022-01-11T13:52:01.092 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2022-01-11T13:52:01.177 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2022-01-11T13:52:01.286 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2022-01-11T13:52:01.373 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2022-01-11T13:52:01.490 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2022-01-11T13:52:01.575 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2022-01-11T13:52:01.682 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2022-01-11T13:52:01.778 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2022-01-11T13:52:01.883 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2022-01-11T13:52:01.979 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2022-01-11T13:52:02.074 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2022-01-11T13:52:02.179 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2022-01-11T13:52:02.275 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2022-01-11T13:52:02.372 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2022-01-11T13:52:02.466 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2022-01-11T13:52:02.563 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2022-01-11T13:52:02.670 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2022-01-11T13:52:02.764 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2022-01-11T13:52:02.869 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2022-01-11T13:52:02.963 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2022-01-11T13:52:03.068 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2022-01-11T13:52:03.151 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2022-01-11T13:52:03.257 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2022-01-11T13:52:03.352 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2022-01-11T13:52:03.458 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2022-01-11T13:52:03.559 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2022-01-11T13:52:03.654 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2022-01-11T13:52:03.748 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2022-01-11T13:52:03.853 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2022-01-11T13:52:03.947 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2022-01-11T13:52:04.052 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2022-01-11T13:52:04.147 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2022-01-11T13:52:04.251 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2022-01-11T13:52:04.346 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2022-01-11T13:52:04.452 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2022-01-11T13:52:04.549 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2022-01-11T13:52:04.655 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2022-01-11T13:52:04.741 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2022-01-11T13:52:04.852 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2022-01-11T13:52:04.938 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2022-01-11T13:52:05.045 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2022-01-11T13:52:05.141 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2022-01-11T13:52:05.248 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2022-01-11T13:52:05.339 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2022-01-11T13:52:05.438 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2022-01-11T13:52:05.535 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2022-01-11T13:52:05.639 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2022-01-11T13:52:05.733 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2022-01-11T13:52:05.837 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2022-01-11T13:52:05.931 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2022-01-11T13:52:06.036 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2022-01-11T13:52:06.132 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2022-01-11T13:52:06.226 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2022-01-11T13:52:06.321 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2022-01-11T13:52:06.425 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2022-01-11T13:52:06.521 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2022-01-11T13:52:06.626 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2022-01-11T13:52:06.720 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2022-01-11T13:52:06.825 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2022-01-11T13:52:06.910 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2022-01-11T13:52:07.014 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2022-01-11T13:52:07.108 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2022-01-11T13:52:07.213 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2022-01-11T13:52:07.307 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2022-01-11T13:52:07.402 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2022-01-11T13:52:07.497 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2022-01-11T13:52:07.603 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2022-01-11T13:52:07.697 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2022-01-11T13:52:07.803 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2022-01-11T13:52:07.888 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2022-01-11T13:52:07.992 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2022-01-11T13:52:08.087 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2022-01-11T13:52:08.191 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2022-01-11T13:52:08.285 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2022-01-11T13:52:08.390 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2022-01-11T13:52:08.474 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2022-01-11T13:52:08.588 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2022-01-11T13:52:08.673 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2022-01-11T13:52:08.788 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2022-01-11T13:52:08.873 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2022-01-11T13:52:08.978 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2022-01-11T13:52:08.980 INFO:teuthology.task.ansible.out: 2022-01-11T13:52:09.064 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2022-01-11T13:52:09.179 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2022-01-11T13:52:09.263 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2022-01-11T13:52:09.368 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2022-01-11T13:52:09.461 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2022-01-11T13:52:09.565 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2022-01-11T13:52:09.649 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2022-01-11T13:52:09.765 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2022-01-11T13:52:09.849 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2022-01-11T13:52:09.954 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2022-01-11T13:52:10.048 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2022-01-11T13:52:10.154 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2022-01-11T13:52:10.249 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2022-01-11T13:52:10.354 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2022-01-11T13:52:10.438 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2022-01-11T13:52:10.554 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2022-01-11T13:52:10.639 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2022-01-11T13:52:10.745 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2022-01-11T13:52:10.840 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2022-01-11T13:52:10.948 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2022-01-11T13:52:11.035 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2022-01-11T13:52:11.139 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2022-01-11T13:52:11.234 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2022-01-11T13:52:11.339 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2022-01-11T13:52:11.434 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2022-01-11T13:52:11.540 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2022-01-11T13:52:11.627 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2022-01-11T13:52:11.733 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2022-01-11T13:52:11.829 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2022-01-11T13:52:11.935 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2022-01-11T13:52:12.020 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2022-01-11T13:52:12.127 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2022-01-11T13:52:12.222 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2022-01-11T13:52:12.327 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2022-01-11T13:52:12.412 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2022-01-11T13:52:12.526 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2022-01-11T13:52:12.610 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2022-01-11T13:52:12.727 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2022-01-11T13:52:12.812 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2022-01-11T13:52:12.928 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2022-01-11T13:52:13.012 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2022-01-11T13:52:13.116 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2022-01-11T13:52:13.200 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2022-01-11T13:52:13.325 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2022-01-11T13:52:13.399 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2022-01-11T13:52:13.514 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2022-01-11T13:52:13.598 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2022-01-11T13:52:13.713 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2022-01-11T13:52:13.798 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2022-01-11T13:52:13.915 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2022-01-11T13:52:13.989 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2022-01-11T13:52:14.116 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2022-01-11T13:52:14.192 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2022-01-11T13:52:14.310 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2022-01-11T13:52:14.384 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2022-01-11T13:52:14.510 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2022-01-11T13:52:14.584 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2022-01-11T13:52:14.701 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2022-01-11T13:52:14.789 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2022-01-11T13:52:14.905 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2022-01-11T13:52:14.990 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2022-01-11T13:52:15.097 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2022-01-11T13:52:15.188 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2022-01-11T13:52:15.338 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2022-01-11T13:52:15.414 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2022-01-11T13:52:15.558 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2022-01-11T13:52:15.634 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2022-01-11T13:52:15.773 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2022-01-11T13:52:15.837 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2022-01-11T13:52:15.964 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2022-01-11T13:52:16.049 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2022-01-11T13:52:16.176 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2022-01-11T13:52:16.251 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2022-01-11T13:52:16.367 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2022-01-11T13:52:16.443 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2022-01-11T13:52:16.570 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2022-01-11T13:52:16.643 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2022-01-11T13:52:16.760 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2022-01-11T13:52:16.833 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2022-01-11T13:52:16.969 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2022-01-11T13:52:17.032 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2022-01-11T13:52:17.158 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2022-01-11T13:52:17.232 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nizamial09@thinkpad 4GoNn989LSUMJ1XNW0dGtA 3f8a1ef7c25bb0491b8e3db6946e69bb0dea077048ff8140382fbae5a02206f7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nizamial09@thinkpad 4GoNn989LSUMJ1XNW0dGtA 3f8a1ef7c25bb0491b8e3db6946e69bb0dea077048ff8140382fbae5a02206f7"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2022-01-11T13:52:17.357 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nizamial09@thinkpad 4GoNn989LSUMJ1XNW0dGtA 3f8a1ef7c25bb0491b8e3db6946e69bb0dea077048ff8140382fbae5a02206f7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nizamial09@thinkpad 4GoNn989LSUMJ1XNW0dGtA 3f8a1ef7c25bb0491b8e3db6946e69bb0dea077048ff8140382fbae5a02206f7"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2022-01-11T13:52:17.431 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2022-01-11T13:52:17.559 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2022-01-11T13:52:17.631 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1238, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "move_home": false, "name": "amctagga", "shell": "/bin/bash", "state": "present", "uid": 1238} 2022-01-11T13:52:17.767 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1238, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "move_home": false, "name": "amctagga", "shell": "/bin/bash", "state": "present", "uid": 1238} 2022-01-11T13:52:17.832 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1239, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "move_home": false, "name": "kevinz", "shell": "/bin/bash", "state": "present", "uid": 1239} 2022-01-11T13:52:17.958 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1239, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "move_home": false, "name": "kevinz", "shell": "/bin/bash", "state": "present", "uid": 1239} 2022-01-11T13:52:18.032 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1240, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "move_home": false, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "uid": 1240} 2022-01-11T13:52:18.157 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1240, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "move_home": false, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "uid": 1240} 2022-01-11T13:52:18.220 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1241, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "move_home": false, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "uid": 1241} 2022-01-11T13:52:18.345 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1241, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "move_home": false, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "uid": 1241} 2022-01-11T13:52:18.421 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1242, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "move_home": false, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "uid": 1242} 2022-01-11T13:52:18.544 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1242, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "move_home": false, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "uid": 1242} 2022-01-11T13:52:18.618 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1243, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "move_home": false, "name": "lflores", "shell": "/bin/bash", "state": "present", "uid": 1243} 2022-01-11T13:52:18.744 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1243, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "move_home": false, "name": "lflores", "shell": "/bin/bash", "state": "present", "uid": 1243} 2022-01-11T13:52:18.818 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1244, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "move_home": false, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "uid": 1244} 2022-01-11T13:52:18.936 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1244, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "move_home": false, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "uid": 1244} 2022-01-11T13:52:19.011 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1245, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "move_home": false, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "uid": 1245} 2022-01-11T13:52:19.138 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1245, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "move_home": false, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "uid": 1245} 2022-01-11T13:52:19.213 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1246, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "move_home": false, "name": "luisrico", "shell": "/bin/bash", "state": "present", "uid": 1246} 2022-01-11T13:52:19.340 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1246, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "move_home": false, "name": "luisrico", "shell": "/bin/bash", "state": "present", "uid": 1246} 2022-01-11T13:52:19.404 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1247, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "move_home": false, "name": "matan", "shell": "/bin/bash", "state": "present", "uid": 1247} 2022-01-11T13:52:19.529 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1247, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "move_home": false, "name": "matan", "shell": "/bin/bash", "state": "present", "uid": 1247} 2022-01-11T13:52:19.603 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1248, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "move_home": false, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "uid": 1248} 2022-01-11T13:52:19.730 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1248, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "move_home": false, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "uid": 1248} 2022-01-11T13:52:19.806 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1249, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "move_home": false, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "uid": 1249} 2022-01-11T13:52:19.923 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1249, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "move_home": false, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "uid": 1249} 2022-01-11T13:52:20.008 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1250, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "move_home": false, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "uid": 1250} 2022-01-11T13:52:20.124 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1250, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "move_home": false, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "uid": 1250} 2022-01-11T13:52:20.208 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1251, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "move_home": false, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "uid": 1251} 2022-01-11T13:52:20.314 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1251, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "move_home": false, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "uid": 1251} 2022-01-11T13:52:20.399 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1252, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "move_home": false, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "uid": 1252} 2022-01-11T13:52:20.516 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1252, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "move_home": false, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "uid": 1252} 2022-01-11T13:52:20.590 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1253, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "move_home": false, "name": "melissali", "shell": "/bin/bash", "state": "present", "uid": 1253} 2022-01-11T13:52:20.705 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1253, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "move_home": false, "name": "melissali", "shell": "/bin/bash", "state": "present", "uid": 1253} 2022-01-11T13:52:22.377 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2022-01-11T13:52:22.378 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2022-01-11T13:52:24.159 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2022-01-11T13:52:24.194 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2022-01-11T13:52:25.707 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2022-01-11T13:52:25.804 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2022-01-11T13:52:27.365 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2022-01-11T13:52:27.430 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2022-01-11T13:52:28.904 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/choffman", "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "name": "choffman", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2022-01-11T13:52:29.030 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/choffman", "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "name": "choffman", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2022-01-11T13:52:30.515 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'oro', 'ovpn': 'oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/oro", "item": {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, "name": "oro", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2022-01-11T13:52:30.665 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'oro', 'ovpn': 'oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/oro", "item": {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, "name": "oro", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2022-01-11T13:52:32.198 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'mrfreezeex', 'ovpn': 'mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/mrfreezeex", "item": {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}, "name": "mrfreezeex", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2022-01-11T13:52:32.327 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:52:32 +0000 (0:00:41.207) 0:00:48.887 ******* 2022-01-11T13:52:32.335 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'mrfreezeex', 'ovpn': 'mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/mrfreezeex", "item": {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}, "name": "mrfreezeex", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2022-01-11T13:52:32.434 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2022-01-11T13:52:32.456 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:52:32 +0000 (0:00:00.128) 0:00:49.016 ******* 2022-01-11T13:52:32.477 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:52:32 +0000 (0:00:00.020) 0:00:49.037 ******* 2022-01-11T13:52:32.725 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:52:32 +0000 (0:00:00.247) 0:00:49.285 ******* 2022-01-11T13:52:32.765 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2022-01-11T13:52:32.767 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@desktop 2022-01-11T13:52:32.769 INFO:teuthology.task.ansible.out: QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2022-01-11T13:52:32.770 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed20 2022-01-11T13:52:32.772 INFO:teuthology.task.ansible.out:82caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name" 2022-01-11T13:52:32.774 INFO:teuthology.task.ansible.out:: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915 2022-01-11T13:52:32.776 INFO:teuthology.task.ansible.out:d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZ 2022-01-11T13:52:32.778 INFO:teuthology.task.ansible.out:YM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a9 2022-01-11T13:52:32.778 INFO:teuthology.task.ansible.out:5579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 2022-01-11T13:52:32.780 INFO:teuthology.task.ansible.out: 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp 2022-01-11T13:52:32.782 INFO:teuthology.task.ansible.out:", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3 2022-01-11T13:52:32.784 INFO:teuthology.task.ansible.out:624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9Jj 2022-01-11T13:52:32.785 INFO:teuthology.task.ansible.out:enDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, 2022-01-11T13:52:32.786 INFO:teuthology.task.ansible.out: {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "sh 2022-01-11T13:52:32.787 INFO:teuthology.task.ansible.out:yamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nizamial09@thinkpad 4GoNn989LSUMJ1XNW0dGtA 2022-01-11T13:52:32.789 INFO:teuthology.task.ansible.out:3f8a1ef7c25bb0491b8e3db6946e69bb0dea077048ff8140382fbae5a02206f7"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", 2022-01-11T13:52:32.791 INFO:teuthology.task.ansible.out:"ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}]}, "changed": false} 2022-01-11T13:52:32.814 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:52:32 +0000 (0:00:00.089) 0:00:49.374 ******* 2022-01-11T13:52:32.825 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@desktop 2022-01-11T13:52:32.827 INFO:teuthology.task.ansible.out: QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2022-01-11T13:52:32.829 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed20 2022-01-11T13:52:32.830 INFO:teuthology.task.ansible.out:82caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name" 2022-01-11T13:52:32.832 INFO:teuthology.task.ansible.out:: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915 2022-01-11T13:52:32.834 INFO:teuthology.task.ansible.out:d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZ 2022-01-11T13:52:32.835 INFO:teuthology.task.ansible.out:YM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a9 2022-01-11T13:52:32.836 INFO:teuthology.task.ansible.out:5579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 2022-01-11T13:52:32.837 INFO:teuthology.task.ansible.out: 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp 2022-01-11T13:52:32.839 INFO:teuthology.task.ansible.out:", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3 2022-01-11T13:52:32.840 INFO:teuthology.task.ansible.out:624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9Jj 2022-01-11T13:52:32.842 INFO:teuthology.task.ansible.out:enDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, 2022-01-11T13:52:32.842 INFO:teuthology.task.ansible.out: {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "sh 2022-01-11T13:52:32.844 INFO:teuthology.task.ansible.out:yamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nizamial09@thinkpad 4GoNn989LSUMJ1XNW0dGtA 2022-01-11T13:52:32.846 INFO:teuthology.task.ansible.out:3f8a1ef7c25bb0491b8e3db6946e69bb0dea077048ff8140382fbae5a02206f7"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", 2022-01-11T13:52:32.847 INFO:teuthology.task.ansible.out:"ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}]}, "changed": false} 2022-01-11T13:52:32.871 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:52:32 +0000 (0:00:00.057) 0:00:49.431 ******* 2022-01-11T13:52:33.681 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2022-01-11T13:52:33.683 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"after": "b39424e053d42ee607f17b88c1886c2d02811d8f", "attempts": 1, "before": "b39424e053d42ee607f17b88c1886c2d02811d8f", "changed": false, "remote_url_changed": false} 2022-01-11T13:52:33.703 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:52:33 +0000 (0:00:00.831) 0:00:50.263 ******* 2022-01-11T13:52:33.710 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"after": "b39424e053d42ee607f17b88c1886c2d02811d8f", "attempts": 1, "before": "b39424e053d42ee607f17b88c1886c2d02811d8f", "changed": false, "remote_url_changed": false} 2022-01-11T13:52:33.734 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:52:33 +0000 (0:00:00.030) 0:00:50.294 ******* 2022-01-11T13:52:34.159 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2022-01-11T13:52:34.161 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2022-01-11T13:52:34.212 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2022-01-11T13:52:34.369 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2022-01-11T13:52:34.432 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2022-01-11T13:52:34.579 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2022-01-11T13:52:34.652 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2022-01-11T13:52:34.789 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2022-01-11T13:52:34.790 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2022-01-11T13:52:34.862 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2022-01-11T13:52:34.864 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2022-01-11T13:52:34.997 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2022-01-11T13:52:35.071 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2022-01-11T13:52:35.207 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2022-01-11T13:52:35.209 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2022-01-11T13:52:35.271 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2022-01-11T13:52:35.273 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2022-01-11T13:52:35.427 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2022-01-11T13:52:35.491 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2022-01-11T13:52:35.628 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2022-01-11T13:52:35.692 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2022-01-11T13:52:35.841 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2022-01-11T13:52:35.852 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:52:35.910 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2022-01-11T13:52:35.921 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:52:36.079 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2022-01-11T13:52:36.081 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2022-01-11T13:52:36.173 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2022-01-11T13:52:36.175 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2022-01-11T13:52:36.329 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2022-01-11T13:52:36.331 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2022-01-11T13:52:36.355 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:52:36.364 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:52:36.396 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2022-01-11T13:52:36.398 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2022-01-11T13:52:36.407 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:52:36.417 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:52:36.579 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2022-01-11T13:52:36.621 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2022-01-11T13:52:36.791 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2022-01-11T13:52:36.793 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2022-01-11T13:52:36.835 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2022-01-11T13:52:36.837 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2022-01-11T13:52:37.002 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2022-01-11T13:52:37.055 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2022-01-11T13:52:37.215 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2022-01-11T13:52:37.260 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2022-01-11T13:52:37.428 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2022-01-11T13:52:37.471 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2022-01-11T13:52:37.639 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2022-01-11T13:52:37.641 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2022-01-11T13:52:37.683 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2022-01-11T13:52:37.685 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2022-01-11T13:52:37.840 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2022-01-11T13:52:37.842 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2022-01-11T13:52:37.903 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2022-01-11T13:52:37.905 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2022-01-11T13:52:38.060 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2022-01-11T13:52:38.113 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2022-01-11T13:52:38.278 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2022-01-11T13:52:38.321 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2022-01-11T13:52:38.496 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlI 2022-01-11T13:52:38.498 INFO:teuthology.task.ansible.out:Qv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2022-01-11T13:52:38.549 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlI 2022-01-11T13:52:38.551 INFO:teuthology.task.ansible.out:Qv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2022-01-11T13:52:38.707 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2022-01-11T13:52:38.761 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2022-01-11T13:52:38.908 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2022-01-11T13:52:38.971 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2022-01-11T13:52:39.118 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2022-01-11T13:52:39.181 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2022-01-11T13:52:39.328 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2022-01-11T13:52:39.402 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2022-01-11T13:52:39.538 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2022-01-11T13:52:39.612 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2022-01-11T13:52:39.749 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2022-01-11T13:52:39.751 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2022-01-11T13:52:39.823 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2022-01-11T13:52:39.825 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2022-01-11T13:52:39.961 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2022-01-11T13:52:40.035 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2022-01-11T13:52:40.172 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2022-01-11T13:52:40.256 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2022-01-11T13:52:40.384 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2022-01-11T13:52:40.458 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2022-01-11T13:52:40.586 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2022-01-11T13:52:40.671 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2022-01-11T13:52:40.799 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2022-01-11T13:52:40.882 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2022-01-11T13:52:41.020 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2022-01-11T13:52:41.095 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2022-01-11T13:52:41.222 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2022-01-11T13:52:41.306 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2022-01-11T13:52:41.422 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2022-01-11T13:52:41.424 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2022-01-11T13:52:41.508 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2022-01-11T13:52:41.509 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2022-01-11T13:52:41.644 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2022-01-11T13:52:41.646 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2022-01-11T13:52:41.718 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2022-01-11T13:52:41.719 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2022-01-11T13:52:41.855 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2022-01-11T13:52:41.918 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2022-01-11T13:52:42.065 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2022-01-11T13:52:42.129 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2022-01-11T13:52:42.266 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2022-01-11T13:52:42.341 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2022-01-11T13:52:42.478 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2022-01-11T13:52:42.479 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2022-01-11T13:52:42.551 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2022-01-11T13:52:42.553 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2022-01-11T13:52:42.689 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2022-01-11T13:52:42.690 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2022-01-11T13:52:42.692 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2022-01-11T13:52:42.763 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2022-01-11T13:52:42.765 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2022-01-11T13:52:42.766 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2022-01-11T13:52:42.899 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2022-01-11T13:52:42.973 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2022-01-11T13:52:43.100 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2022-01-11T13:52:43.184 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2022-01-11T13:52:43.311 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2022-01-11T13:52:43.407 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2022-01-11T13:52:43.523 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2022-01-11T13:52:43.618 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2022-01-11T13:52:43.726 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2022-01-11T13:52:43.820 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2022-01-11T13:52:43.941 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2022-01-11T13:52:43.942 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2022-01-11T13:52:44.025 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2022-01-11T13:52:44.027 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2022-01-11T13:52:44.161 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2022-01-11T13:52:44.245 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2022-01-11T13:52:44.360 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2022-01-11T13:52:44.455 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2022-01-11T13:52:44.570 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2022-01-11T13:52:44.572 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2022-01-11T13:52:44.654 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2022-01-11T13:52:44.656 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2022-01-11T13:52:44.780 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2022-01-11T13:52:44.864 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2022-01-11T13:52:44.980 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2022-01-11T13:52:45.074 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2022-01-11T13:52:45.209 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2022-01-11T13:52:45.211 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2022-01-11T13:52:45.282 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2022-01-11T13:52:45.284 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2022-01-11T13:52:45.408 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2022-01-11T13:52:45.482 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2022-01-11T13:52:45.619 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2022-01-11T13:52:45.694 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2022-01-11T13:52:45.820 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2022-01-11T13:52:45.914 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2022-01-11T13:52:46.029 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2022-01-11T13:52:46.114 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2022-01-11T13:52:46.230 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2022-01-11T13:52:46.325 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2022-01-11T13:52:46.440 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2022-01-11T13:52:46.533 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2022-01-11T13:52:46.649 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2022-01-11T13:52:46.733 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2022-01-11T13:52:46.849 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2022-01-11T13:52:46.944 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2022-01-11T13:52:47.061 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2022-01-11T13:52:47.146 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2022-01-11T13:52:47.271 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2022-01-11T13:52:47.355 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2022-01-11T13:52:47.480 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2022-01-11T13:52:47.564 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2022-01-11T13:52:47.679 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2022-01-11T13:52:47.763 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2022-01-11T13:52:47.892 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2022-01-11T13:52:47.976 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2022-01-11T13:52:48.102 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2022-01-11T13:52:48.186 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2022-01-11T13:52:48.323 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2022-01-11T13:52:48.396 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2022-01-11T13:52:48.533 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2022-01-11T13:52:48.606 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2022-01-11T13:52:48.732 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2022-01-11T13:52:48.816 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2022-01-11T13:52:48.941 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2022-01-11T13:52:49.024 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2022-01-11T13:52:49.150 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2022-01-11T13:52:49.234 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2022-01-11T13:52:49.360 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2022-01-11T13:52:49.443 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2022-01-11T13:52:49.568 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2022-01-11T13:52:49.570 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2022-01-11T13:52:49.651 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2022-01-11T13:52:49.653 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2022-01-11T13:52:49.776 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2022-01-11T13:52:49.863 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2022-01-11T13:52:49.979 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2022-01-11T13:52:50.063 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2022-01-11T13:52:50.191 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2022-01-11T13:52:50.275 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2022-01-11T13:52:50.400 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2022-01-11T13:52:50.402 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2022-01-11T13:52:50.484 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2022-01-11T13:52:50.486 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2022-01-11T13:52:50.610 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2022-01-11T13:52:50.693 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2022-01-11T13:52:50.818 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2022-01-11T13:52:50.902 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2022-01-11T13:52:51.028 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2022-01-11T13:52:51.101 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2022-01-11T13:52:51.238 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2022-01-11T13:52:51.311 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2022-01-11T13:52:51.448 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2022-01-11T13:52:51.521 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2022-01-11T13:52:51.656 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2022-01-11T13:52:51.729 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2022-01-11T13:52:51.875 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2022-01-11T13:52:51.931 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2022-01-11T13:52:52.077 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2022-01-11T13:52:52.079 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2022-01-11T13:52:52.140 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2022-01-11T13:52:52.142 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2022-01-11T13:52:52.286 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2022-01-11T13:52:52.350 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2022-01-11T13:52:52.498 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2022-01-11T13:52:52.550 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2022-01-11T13:52:52.707 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2022-01-11T13:52:52.761 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2022-01-11T13:52:52.908 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2022-01-11T13:52:52.971 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2022-01-11T13:52:53.108 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2022-01-11T13:52:53.182 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2022-01-11T13:52:53.318 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2022-01-11T13:52:53.381 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2022-01-11T13:52:53.526 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2022-01-11T13:52:53.528 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2022-01-11T13:52:53.590 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2022-01-11T13:52:53.592 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2022-01-11T13:52:53.726 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2022-01-11T13:52:53.789 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2022-01-11T13:52:53.937 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2022-01-11T13:52:54.000 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2022-01-11T13:52:54.147 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2022-01-11T13:52:54.212 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2022-01-11T13:52:54.359 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2022-01-11T13:52:54.423 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2022-01-11T13:52:54.559 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2022-01-11T13:52:54.644 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2022-01-11T13:52:54.769 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2022-01-11T13:52:54.855 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2022-01-11T13:52:54.970 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2022-01-11T13:52:55.053 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2022-01-11T13:52:55.179 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2022-01-11T13:52:55.263 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2022-01-11T13:52:55.389 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2022-01-11T13:52:55.472 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2022-01-11T13:52:55.610 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2022-01-11T13:52:55.683 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2022-01-11T13:52:55.821 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2022-01-11T13:52:55.896 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2022-01-11T13:52:56.037 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2022-01-11T13:52:56.112 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2022-01-11T13:52:56.250 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2022-01-11T13:52:56.325 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2022-01-11T13:52:56.461 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2022-01-11T13:52:56.525 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2022-01-11T13:52:56.662 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2022-01-11T13:52:56.725 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2022-01-11T13:52:56.881 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2022-01-11T13:52:56.944 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2022-01-11T13:52:57.080 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2022-01-11T13:52:57.154 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2022-01-11T13:52:57.291 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2022-01-11T13:52:57.354 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2022-01-11T13:52:57.501 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2022-01-11T13:52:57.503 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2022-01-11T13:52:57.564 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2022-01-11T13:52:57.566 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2022-01-11T13:52:57.704 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2022-01-11T13:52:57.768 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2022-01-11T13:52:57.904 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2022-01-11T13:52:57.983 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2022-01-11T13:52:58.118 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2022-01-11T13:52:58.193 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2022-01-11T13:52:58.319 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2022-01-11T13:52:58.383 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2022-01-11T13:52:58.521 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2022-01-11T13:52:58.594 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2022-01-11T13:52:58.732 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2022-01-11T13:52:58.806 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2022-01-11T13:52:58.942 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2022-01-11T13:52:59.016 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2022-01-11T13:52:59.154 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2022-01-11T13:52:59.217 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2022-01-11T13:52:59.365 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2022-01-11T13:52:59.438 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2022-01-11T13:52:59.566 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2022-01-11T13:52:59.641 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2022-01-11T13:52:59.800 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2022-01-11T13:52:59.855 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2022-01-11T13:53:00.012 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2022-01-11T13:53:00.067 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2022-01-11T13:53:00.214 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4m0LAcC7XiFiXFtYcCvZqX1CtyV9JU+VUEawZJpWrGTOr8c+vIexq7+WKg+eIUMot4+dtfwG8lByS1VofglIe5/quv17Xj7dO/hF2WkV8O7RYt7NhzmrKg54yB73pbRpvqzGceWWDvGRsWQGzbY7ARqOp4FPIDJvgXABCkak+hmjXyXpvXN418CBDbDmxHWwFb9QtB/MpQFiFrfl1EMDj8DaXZNWgo5J/qytic6LgiyPHhHfhSylk4cl9TGftNatobpMjqBT0O2/plk/qf+7pHPuvGSgihjijdpEPzVfctXE2OmsQAIZgkzXeCqts+s6ZYdUKq1FvpV5rMM8ZLqfCkDjEzFENJS68YYE2EjxAEgMzj7h/sWc4B5I5pxkAqVAQSGjYRu6rCCbFz8PXUdTyW+Vs1YeylnlBMiCa4xE4gwvW3HlWEnumtyURd1PRENrGJsJ/0uX6bpltbSg4sGACRYpLPJMm3OhqI3Jm/J/e2NLjfCw+svU2447xnfJOh68= pdhange@vaquita\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoeyVaMjhcGrWQB+4Rq+SSIi8+OY/kx2s1vofiGt1gu3RNtFmG0Df1VITJJ32dbzaW7kTNxaRvhldNd/n7yEMssG+Wg+a5WKvSudlbVMmscbSXdFuVbcCW1jNrTGCYrxL7Sn5e7NQEQfAsCbfXwTcYBw33NdBVovC+gs2V3sZGvX3Y0akZONNuVg1mhu0L4m1/5IegJh005KeilwetHA2A5Dr10RG6oyIgT7t4z5gZXsrOiG3UQBXaekILYV9jxMWoFXGkVLc8TpL9SUAdxE3Owc0u1XiPqYYlEN+zDvE0554Zj8Oc2ZO6HRXCZXQFvzXeCD12YswcOfkIDmJRF7GT prashant@mauli", "key_options": null, "k 2022-01-11T13:53:00.216 INFO:teuthology.task.ansible.out:eyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2022-01-11T13:53:00.277 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4m0LAcC7XiFiXFtYcCvZqX1CtyV9JU+VUEawZJpWrGTOr8c+vIexq7+WKg+eIUMot4+dtfwG8lByS1VofglIe5/quv17Xj7dO/hF2WkV8O7RYt7NhzmrKg54yB73pbRpvqzGceWWDvGRsWQGzbY7ARqOp4FPIDJvgXABCkak+hmjXyXpvXN418CBDbDmxHWwFb9QtB/MpQFiFrfl1EMDj8DaXZNWgo5J/qytic6LgiyPHhHfhSylk4cl9TGftNatobpMjqBT0O2/plk/qf+7pHPuvGSgihjijdpEPzVfctXE2OmsQAIZgkzXeCqts+s6ZYdUKq1FvpV5rMM8ZLqfCkDjEzFENJS68YYE2EjxAEgMzj7h/sWc4B5I5pxkAqVAQSGjYRu6rCCbFz8PXUdTyW+Vs1YeylnlBMiCa4xE4gwvW3HlWEnumtyURd1PRENrGJsJ/0uX6bpltbSg4sGACRYpLPJMm3OhqI3Jm/J/e2NLjfCw+svU2447xnfJOh68= pdhange@vaquita\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoeyVaMjhcGrWQB+4Rq+SSIi8+OY/kx2s1vofiGt1gu3RNtFmG0Df1VITJJ32dbzaW7kTNxaRvhldNd/n7yEMssG+Wg+a5WKvSudlbVMmscbSXdFuVbcCW1jNrTGCYrxL7Sn5e7NQEQfAsCbfXwTcYBw33NdBVovC+gs2V3sZGvX3Y0akZONNuVg1mhu0L4m1/5IegJh005KeilwetHA2A5Dr10RG6oyIgT7t4z5gZXsrOiG3UQBXaekILYV9jxMWoFXGkVLc8TpL9SUAdxE3Owc0u1XiPqYYlEN+zDvE0554Zj8Oc2ZO6HRXCZXQFvzXeCD12YswcOfkIDmJRF7GT prashant@mauli", "key_options": null, "k 2022-01-11T13:53:00.279 INFO:teuthology.task.ansible.out:eyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2022-01-11T13:53:00.424 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2022-01-11T13:53:00.487 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2022-01-11T13:53:00.622 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2022-01-11T13:53:00.822 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2022-01-11T13:53:00.833 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2022-01-11T13:53:00.835 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2022-01-11T13:53:00.928 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2022-01-11T13:53:00.929 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2022-01-11T13:53:01.033 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nizamial09@thinkpad 4GoNn989LSUMJ1XNW0dGtA 3f8a1ef7c25bb0491b8e3db6946e69bb0dea077048ff8140382fbae5a02206f7'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nizamial09@thinkpad 4GoNn989LSUMJ1XNW0dGtA 3f8a1ef7c25bb0491b8e3db6946e69bb0dea077048ff8140382fbae5a02206f7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC++MYddG6qYt7fXIvbaR1IXivOySDR//T+sQR3iRtmDQHZGPGiWshltpzeCboa51qeXurN8JYIYq1ZqvLEaOvwbEMtSLthPNl+FZMfe8E4YsuS3212fgbPhEwEBYo+7QZThCseDCewnE5Iz2I/tqlT69T10QZgbyHoZzT9aTPQBQHMF8by7ze9+V+cTD9ZrTG+fMIostxhJAOC+AsTGXwmtcWWrmZXKm/+2g7FTicZyCujXVzXY01c6rJhi1anRQh3pdfm+e6pmf37wHpPunHY2Xhiuy88o5WrzxKcLDUavwI+NI0+la54c0Qiv52LZPmDvPpbNh/7zmCu2vWHVayz1DZb52K5vP1Lqb7oD/cFfQEZ8m4UqzmtemT4gowJ/hFMh8RKPxCho3DlRAw4a7EhBvGiTgyvlcCKVk3S0aeIhvXsewWjOoRNHHvxbvKRCm2b0kSfnDnsZYyAsA8QpTbZYTmbbXHdyLLBS8gfFtB0aDPOPb2Nry2+ic/3hrIY5/c= nizamudeena@fedora", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2022-01-11T13:53:01.138 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nizamial09@thinkpad 4GoNn989LSUMJ1XNW0dGtA 3f8a1ef7c25bb0491b8e3db6946e69bb0dea077048ff8140382fbae5a02206f7'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nizamial09@thinkpad 4GoNn989LSUMJ1XNW0dGtA 3f8a1ef7c25bb0491b8e3db6946e69bb0dea077048ff8140382fbae5a02206f7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC++MYddG6qYt7fXIvbaR1IXivOySDR//T+sQR3iRtmDQHZGPGiWshltpzeCboa51qeXurN8JYIYq1ZqvLEaOvwbEMtSLthPNl+FZMfe8E4YsuS3212fgbPhEwEBYo+7QZThCseDCewnE5Iz2I/tqlT69T10QZgbyHoZzT9aTPQBQHMF8by7ze9+V+cTD9ZrTG+fMIostxhJAOC+AsTGXwmtcWWrmZXKm/+2g7FTicZyCujXVzXY01c6rJhi1anRQh3pdfm+e6pmf37wHpPunHY2Xhiuy88o5WrzxKcLDUavwI+NI0+la54c0Qiv52LZPmDvPpbNh/7zmCu2vWHVayz1DZb52K5vP1Lqb7oD/cFfQEZ8m4UqzmtemT4gowJ/hFMh8RKPxCho3DlRAw4a7EhBvGiTgyvlcCKVk3S0aeIhvXsewWjOoRNHHvxbvKRCm2b0kSfnDnsZYyAsA8QpTbZYTmbbXHdyLLBS8gfFtB0aDPOPb2Nry2+ic/3hrIY5/c= nizamudeena@fedora", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2022-01-11T13:53:01.243 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2022-01-11T13:53:01.349 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2022-01-11T13:53:01.453 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2022-01-11T13:53:01.557 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2022-01-11T13:53:01.661 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2022-01-11T13:53:01.766 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2022-01-11T13:53:01.871 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2022-01-11T13:53:01.966 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2022-01-11T13:53:02.074 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2022-01-11T13:53:02.179 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2022-01-11T13:53:02.285 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2022-01-11T13:53:02.382 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2022-01-11T13:53:02.486 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2022-01-11T13:53:02.591 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2022-01-11T13:53:02.697 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2022-01-11T13:53:02.792 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2022-01-11T13:53:02.898 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2022-01-11T13:53:03.004 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2022-01-11T13:53:03.110 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2022-01-11T13:53:03.215 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2022-01-11T13:53:03.320 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2022-01-11T13:53:03.425 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2022-01-11T13:53:03.531 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2022-01-11T13:53:03.626 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2022-01-11T13:53:03.731 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2022-01-11T13:53:03.836 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2022-01-11T13:53:03.941 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2022-01-11T13:53:04.047 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2022-01-11T13:53:04.142 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2022-01-11T13:53:04.267 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2022-01-11T13:53:04.362 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2022-01-11T13:53:04.466 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2022-01-11T13:53:04.571 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2022-01-11T13:53:04.675 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2022-01-11T13:53:04.780 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2022-01-11T13:53:04.875 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2022-01-11T13:53:04.990 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2022-01-11T13:53:05.085 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2022-01-11T13:53:05.190 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2022-01-11T13:53:05.294 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2022-01-11T13:53:05.409 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2022-01-11T13:53:05.502 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2022-01-11T13:53:05.606 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7LPHiaPQXdE+yt9cn7UMuRr8k/LjSUmMrnrAqp8ySQtX6OA3/NcOsxg6FJ45chZqeoXwhsYEW+7pTInDqFlOlmVynR26lWnZfk9DjBJZcloUCn8y5MR9QitNYsOMs5KwCEFmBH6sP+2vCFocXBnKQcVZLRfip3MEG4LoWYfI4ir7fLmK1Hr5G+GhtMstXvVSG4VE4IgbU9/LRIOv7P9qFn4bRJoUSiS2dDbUxVIuneA9bIEnB9ZEVGahEzlUhx6qDSs0FvrnneCQ1ayOFpxClzjT5GuKk/J3LJvWAw8CAwDXCf3FzsWQsvb5ru5VSazUI9uA3o9wWkbLW2oZPgtzI8lmwbL+MT58T92tqaJOPID3IZDGjzNrRqU5BQYpvFYHpHRRiqV625og+ktYfXgO+dTbgAxScomqutydI4cyJusDtnWtljWnEocS+2hUSZpmB6B3Wj70TuuzsbYrcrJ/NgRpQNX4dwHCJl6vPT+xkoLh1imYAC7klf7cnrOyGcs= choffman@choffman.net", "key_options": null, "keyfile": "/home/choffman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "choffman", "validate_certs": true} 2022-01-11T13:53:05.710 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7LPHiaPQXdE+yt9cn7UMuRr8k/LjSUmMrnrAqp8ySQtX6OA3/NcOsxg6FJ45chZqeoXwhsYEW+7pTInDqFlOlmVynR26lWnZfk9DjBJZcloUCn8y5MR9QitNYsOMs5KwCEFmBH6sP+2vCFocXBnKQcVZLRfip3MEG4LoWYfI4ir7fLmK1Hr5G+GhtMstXvVSG4VE4IgbU9/LRIOv7P9qFn4bRJoUSiS2dDbUxVIuneA9bIEnB9ZEVGahEzlUhx6qDSs0FvrnneCQ1ayOFpxClzjT5GuKk/J3LJvWAw8CAwDXCf3FzsWQsvb5ru5VSazUI9uA3o9wWkbLW2oZPgtzI8lmwbL+MT58T92tqaJOPID3IZDGjzNrRqU5BQYpvFYHpHRRiqV625og+ktYfXgO+dTbgAxScomqutydI4cyJusDtnWtljWnEocS+2hUSZpmB6B3Wj70TuuzsbYrcrJ/NgRpQNX4dwHCJl6vPT+xkoLh1imYAC7klf7cnrOyGcs= choffman@choffman.net", "key_options": null, "keyfile": "/home/choffman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "choffman", "validate_certs": true} 2022-01-11T13:53:05.817 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'oro', 'ovpn': 'oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAqZvQXoWrH8ZaQ9Y8KK1B9JBCuD8Zspp2mWtjrWXGotKBCSGdDn3QSDEFtwpoK6vXZ3BijhUmrQ1bNUSo59556+T+wpfJAPJjuvB9i7A2lKSX74heuDl0QhJwHdt9mbCtpZ//7+IU0srzTn5qKwJ16GXeomgcyOZmefSGi/sbn0wRUR/UGA/LOJ3EK0nXcxyrv3iDfv75KRL87kaXtPRJRvSlXB5PSuDQtIoXhzMuD4nnh879Gwy0sYE/FUXcbl7LQb+/9rNJLE2POVNjArGv7RMp3y0K3k2/02AjbZvQwhfuIVayzTpmUiGY2Fo6hJRkH8ZT0aYF8uotfweGK18EYw== rsa-key-20151116", "key_options": null, "keyfile": "/home/oro/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oro", "validate_certs": true} 2022-01-11T13:53:05.922 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'oro', 'ovpn': 'oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAqZvQXoWrH8ZaQ9Y8KK1B9JBCuD8Zspp2mWtjrWXGotKBCSGdDn3QSDEFtwpoK6vXZ3BijhUmrQ1bNUSo59556+T+wpfJAPJjuvB9i7A2lKSX74heuDl0QhJwHdt9mbCtpZ//7+IU0srzTn5qKwJ16GXeomgcyOZmefSGi/sbn0wRUR/UGA/LOJ3EK0nXcxyrv3iDfv75KRL87kaXtPRJRvSlXB5PSuDQtIoXhzMuD4nnh879Gwy0sYE/FUXcbl7LQb+/9rNJLE2POVNjArGv7RMp3y0K3k2/02AjbZvQwhfuIVayzTpmUiGY2Fo6hJRkH8ZT0aYF8uotfweGK18EYw== rsa-key-20151116", "key_options": null, "keyfile": "/home/oro/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oro", "validate_certs": true} 2022-01-11T13:53:06.032 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'mrfreezeex', 'ovpn': 'mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIADWYnXrwweaWs8/3gaUDctlaelfSPRh0Cs3G2SHbER0 mrfreezeex@inferno", "key_options": null, "keyfile": "/home/mrfreezeex/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mrfreezeex", "validate_certs": true} 2022-01-11T13:53:06.150 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:06 +0000 (0:00:32.416) 0:01:22.710 ******* 2022-01-11T13:53:06.157 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'name': 'mrfreezeex', 'ovpn': 'mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIADWYnXrwweaWs8/3gaUDctlaelfSPRh0Cs3G2SHbER0 mrfreezeex@inferno", "key_options": null, "keyfile": "/home/mrfreezeex/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mrfreezeex", "validate_certs": true} 2022-01-11T13:53:06.238 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:06 +0000 (0:00:00.088) 0:01:22.798 ******* 2022-01-11T13:53:06.247 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2022-01-11T13:53:06.249 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.263 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.274 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.286 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.297 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.308 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.319 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.321 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.332 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.343 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.354 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.356 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.366 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.377 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.379 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.388 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.556 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2022-01-11T13:53:06.558 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.567 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.610 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2022-01-11T13:53:06.622 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.624 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:06.770 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2022-01-11T13:53:06.834 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2022-01-11T13:53:07.016 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2022-01-11T13:53:07.027 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.038 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.049 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.050 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.071 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.082 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.093 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.095 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.104 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2022-01-11T13:53:07.106 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.116 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.127 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.129 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.140 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.142 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.156 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.157 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.159 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.170 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.170 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.182 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.182 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.183 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.196 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.199 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.207 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.209 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.222 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c 2022-01-11T13:53:07.224 INFO:teuthology.task.ansible.out:94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.234 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.236 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.247 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.248 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.260 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.281 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.283 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff0998182244 2022-01-11T13:53:07.284 INFO:teuthology.task.ansible.out:55a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.295 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.296 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.306 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.307 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.318 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.319 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.329 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.331 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.342 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.343 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.343 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.356 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.358 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.360 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.371 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.372 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.385 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.386 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.388 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.390 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.391 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.402 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.406 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.415 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.417 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.427 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.429 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.439 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.441 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.451 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.453 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.463 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.464 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.464 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.475 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.477 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.487 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.489 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.499 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.500 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.501 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.512 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.513 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.524 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.525 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.536 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.538 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.548 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.550 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.560 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.562 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.571 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.573 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/H 2022-01-11T13:53:07.574 INFO:teuthology.task.ansible.out:j/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.583 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.585 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.596 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.598 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.607 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.609 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.620 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.622 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.631 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.633 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.643 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.645 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.656 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.657 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.668 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.669 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.681 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.683 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.694 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.706 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.707 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.717 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.718 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.719 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.719 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZN 2022-01-11T13:53:07.720 INFO:teuthology.task.ansible.out:IKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.731 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.732 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.744 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.755 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.756 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.768 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.770 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.780 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.782 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.792 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.794 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.794 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.805 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.806 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.816 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.818 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.828 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.830 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.840 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.842 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.852 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.854 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.864 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.876 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.877 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.888 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.890 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.901 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.905 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.913 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.915 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.926 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.929 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nizamial09@thinkpad 4GoNn989LSUMJ1XNW0dGtA 3f8a1ef7c25bb0491b8e3db6946e69bb0dea077048ff8140382fbae5a02206f7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nizamial09@thinkpad 4GoNn989LSUMJ1XNW0dGtA 3f8a1ef7c25bb0491b8e3db6946e69bb0dea077048ff8140382fbae5a02206f7"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.940 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.950 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.952 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.963 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.964 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.966 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nizamial09@thinkpad 4GoNn989LSUMJ1XNW0dGtA 3f8a1ef7c25bb0491b8e3db6946e69bb0dea077048ff8140382fbae5a02206f7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nizamial09@thinkpad 4GoNn989LSUMJ1XNW0dGtA 3f8a1ef7c25bb0491b8e3db6946e69bb0dea077048ff8140382fbae5a02206f7"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.966 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "c 2022-01-11T13:53:07.967 INFO:teuthology.task.ansible.out:hanged": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.982 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.989 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:07.991 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.002 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.005 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.016 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.018 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.027 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.029 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.039 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.041 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.041 INFO:teuthology.task.ansible.out: 2022-01-11T13:53:08.053 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.054 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.065 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.067 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.077 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.078 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.091 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'oro', 'ovpn': 'oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.103 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.105 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mrfreezeex', 'ovpn': 'mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.124 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.127 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'oro', 'ovpn': 'oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oro", "ovpn": "oro@oro-ubuntu-bm r1PDxMqo/AP6wFx5ZWQ3eg 18bf1a1177da6dc89958adf65a4813a7da3e899cd94dc86384240ea3b5e00361"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.146 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:08 +0000 (0:00:01.907) 0:01:24.706 ******* 2022-01-11T13:53:08.153 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'name': 'mrfreezeex', 'ovpn': 'mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mrfreezeex", "ovpn": "mrfreezeex@inferno ade2FH1BYDNTWeREZEJpFQ 32d052805908bc929983119748701a6c882cfbd2dd32c511bdcfbe6d32efb7ef"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.197 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:08 +0000 (0:00:00.050) 0:01:24.757 ******* 2022-01-11T13:53:08.214 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2022-01-11T13:53:08.216 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.237 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:08 +0000 (0:00:00.039) 0:01:24.797 ******* 2022-01-11T13:53:08.264 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:08.289 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:08 +0000 (0:00:00.051) 0:01:24.848 ******* 2022-01-11T13:53:08.514 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2022-01-11T13:53:08.516 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2022-01-11T13:53:08.567 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2022-01-11T13:53:08.705 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2022-01-11T13:53:08.757 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2022-01-11T13:53:08.904 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2022-01-11T13:53:08.947 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2022-01-11T13:53:09.094 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2022-01-11T13:53:09.146 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2022-01-11T13:53:09.282 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2022-01-11T13:53:09.335 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2022-01-11T13:53:09.481 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2022-01-11T13:53:09.523 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2022-01-11T13:53:09.670 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2022-01-11T13:53:09.723 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2022-01-11T13:53:09.858 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2022-01-11T13:53:09.911 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2022-01-11T13:53:10.057 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2022-01-11T13:53:10.110 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2022-01-11T13:53:10.255 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2022-01-11T13:53:10.298 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2022-01-11T13:53:10.444 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2022-01-11T13:53:10.486 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2022-01-11T13:53:10.634 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2022-01-11T13:53:10.677 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2022-01-11T13:53:10.822 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2022-01-11T13:53:10.874 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2022-01-11T13:53:11.020 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2022-01-11T13:53:11.062 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2022-01-11T13:53:11.209 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2022-01-11T13:53:11.263 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2022-01-11T13:53:11.410 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2022-01-11T13:53:11.453 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2022-01-11T13:53:11.610 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2022-01-11T13:53:11.643 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2022-01-11T13:53:11.800 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2022-01-11T13:53:11.842 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2022-01-11T13:53:11.988 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2022-01-11T13:53:12.040 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2022-01-11T13:53:12.180 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2022-01-11T13:53:12.284 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2022-01-11T13:53:12.379 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2022-01-11T13:53:12.422 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2022-01-11T13:53:12.568 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2022-01-11T13:53:12.621 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2022-01-11T13:53:12.758 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2022-01-11T13:53:12.810 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2022-01-11T13:53:12.958 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2022-01-11T13:53:13.001 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2022-01-11T13:53:13.148 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2022-01-11T13:53:13.191 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2022-01-11T13:53:13.337 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2022-01-11T13:53:13.390 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2022-01-11T13:53:13.537 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2022-01-11T13:53:13.580 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2022-01-11T13:53:13.728 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2022-01-11T13:53:13.780 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2022-01-11T13:53:13.926 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2022-01-11T13:53:13.968 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2022-01-11T13:53:14.124 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2022-01-11T13:53:14.167 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2022-01-11T13:53:14.313 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2022-01-11T13:53:14.356 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2022-01-11T13:53:14.513 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2022-01-11T13:53:14.555 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2022-01-11T13:53:14.703 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2022-01-11T13:53:14.746 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2022-01-11T13:53:14.898 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2022-01-11T13:53:14.941 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2022-01-11T13:53:15.087 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2022-01-11T13:53:15.140 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2022-01-11T13:53:15.289 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2022-01-11T13:53:15.324 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2022-01-11T13:53:15.482 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2022-01-11T13:53:15.514 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2022-01-11T13:53:15.670 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2022-01-11T13:53:15.713 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2022-01-11T13:53:15.859 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2022-01-11T13:53:15.912 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2022-01-11T13:53:17.417 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2022-01-11T13:53:17.438 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2022-01-11T13:53:18.913 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2022-01-11T13:53:19.008 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2022-01-11T13:53:20.389 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2022-01-11T13:53:20.504 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2022-01-11T13:53:21.893 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2022-01-11T13:53:22.019 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2022-01-11T13:53:23.423 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2022-01-11T13:53:23.571 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2022-01-11T13:53:24.860 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2022-01-11T13:53:25.061 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2022-01-11T13:53:25.072 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2022-01-11T13:53:25.272 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2022-01-11T13:53:25.274 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2022-01-11T13:53:25.460 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2022-01-11T13:53:25.482 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2022-01-11T13:53:25.650 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2022-01-11T13:53:25.671 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2022-01-11T13:53:25.849 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2022-01-11T13:53:25.861 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2022-01-11T13:53:26.039 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2022-01-11T13:53:26.061 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2022-01-11T13:53:26.231 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2022-01-11T13:53:26.252 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2022-01-11T13:53:26.429 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2022-01-11T13:53:26.441 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2022-01-11T13:53:26.619 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2022-01-11T13:53:26.630 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2022-01-11T13:53:26.819 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2022-01-11T13:53:26.843 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2022-01-11T13:53:26.867 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:26 +0000 (0:00:18.578) 0:01:43.427 ******* 2022-01-11T13:53:27.023 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2022-01-11T13:53:27.078 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:27 +0000 (0:00:00.211) 0:01:43.639 ******* 2022-01-11T13:53:27.416 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2022-01-11T13:53:27.419 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "checksum": "080c38644b4bda11c886c07a56ac73cb2edfd0d3", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "10f9e32e8b4d88eeb5766f85e234cf69", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1641909206.9639122-4011-20190886233430/source", "state": "file", "uid": 0} 2022-01-11T13:53:27.440 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:27 +0000 (0:00:00.360) 0:01:43.999 ******* 2022-01-11T13:53:27.509 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2022-01-11T13:53:27.512 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => { "msg": "Host smithi172.front.sepia.ceph.com is running CentOS 8.3 (n/a)" } 2022-01-11T13:53:27.532 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:27 +0000 (0:00:00.092) 0:01:44.092 ******* 2022-01-11T13:53:27.613 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2022-01-11T13:53:27.615 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2022-01-11T13:53:27.636 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:27 +0000 (0:00:00.103) 0:01:44.196 ******* 2022-01-11T13:53:27.644 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "checksum": "080c38644b4bda11c886c07a56ac73cb2edfd0d3", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "10f9e32e8b4d88eeb5766f85e234cf69", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1641909207.1699018-4043-156910751364433/source", "state": "file", "uid": 0} 2022-01-11T13:53:27.670 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:27 +0000 (0:00:00.033) 0:01:44.230 ******* 2022-01-11T13:53:27.709 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2022-01-11T13:53:27.711 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:27.733 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:27 +0000 (0:00:00.063) 0:01:44.293 ******* 2022-01-11T13:53:27.734 INFO:teuthology.task.ansible.out: 2022-01-11T13:53:27.750 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2022-01-11T13:53:27.752 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => { "msg": "Host smithi161.front.sepia.ceph.com is running CentOS 8.3 (n/a)" } 2022-01-11T13:53:27.774 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:27 +0000 (0:00:00.041) 0:01:44.334 ******* 2022-01-11T13:53:27.871 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2022-01-11T13:53:27.872 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2022-01-11T13:53:27.893 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:27 +0000 (0:00:00.119) 0:01:44.454 ******* 2022-01-11T13:53:27.984 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.007 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.113) 0:01:44.567 ******* 2022-01-11T13:53:28.057 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.082985", "end": "2022-01-11 13:53:28.032952", "rc": 0, "start": "2022-01-11 13:53:27.949967", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2022-01-11T13:53:28.079 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.072) 0:01:44.639 ******* 2022-01-11T13:53:28.170 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.193 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.114) 0:01:44.753 ******* 2022-01-11T13:53:28.274 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.300 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.106) 0:01:44.860 ******* 2022-01-11T13:53:28.309 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.062448", "end": "2022-01-11 13:53:28.281686", "rc": 0, "start": "2022-01-11 13:53:28.219238", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2022-01-11T13:53:28.333 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.033) 0:01:44.893 ******* 2022-01-11T13:53:28.373 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.401 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.067) 0:01:44.961 ******* 2022-01-11T13:53:28.411 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2022-01-11T13:53:28.413 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.437 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.035) 0:01:44.997 ******* 2022-01-11T13:53:28.466 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2022-01-11T13:53:28.468 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.491 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.054) 0:01:45.051 ******* 2022-01-11T13:53:28.510 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2022-01-11T13:53:28.511 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.534 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.042) 0:01:45.094 ******* 2022-01-11T13:53:28.574 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.597 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.063) 0:01:45.157 ******* 2022-01-11T13:53:28.615 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.643 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.043) 0:01:45.201 ******* 2022-01-11T13:53:28.680 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2022-01-11T13:53:28.682 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.704 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.063) 0:01:45.264 ******* 2022-01-11T13:53:28.721 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2022-01-11T13:53:28.723 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.745 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.041) 0:01:45.305 ******* 2022-01-11T13:53:28.783 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2022-01-11T13:53:28.785 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2022-01-11T13:53:28.808 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.062) 0:01:45.368 ******* 2022-01-11T13:53:28.825 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2022-01-11T13:53:28.827 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.850 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.041) 0:01:45.410 ******* 2022-01-11T13:53:28.878 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2022-01-11T13:53:28.880 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.902 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.052) 0:01:45.462 ******* 2022-01-11T13:53:28.931 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2022-01-11T13:53:28.932 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.954 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.051) 0:01:45.514 ******* 2022-01-11T13:53:28.971 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2022-01-11T13:53:28.973 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:28.998 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:28 +0000 (0:00:00.043) 0:01:45.558 ******* 2022-01-11T13:53:29.038 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi161.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2022-01-11T13:53:29.040 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2022-01-11T13:53:29.066 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.067) 0:01:45.626 ******* 2022-01-11T13:53:29.074 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2022-01-11T13:53:29.075 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:29.099 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.033) 0:01:45.659 ******* 2022-01-11T13:53:29.139 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:29.163 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.063) 0:01:45.723 ******* 2022-01-11T13:53:29.181 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2022-01-11T13:53:29.183 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2022-01-11T13:53:29.204 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.041) 0:01:45.764 ******* 2022-01-11T13:53:29.231 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2022-01-11T13:53:29.234 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:29.256 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.051) 0:01:45.816 ******* 2022-01-11T13:53:29.296 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2022-01-11T13:53:29.298 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:29.332 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.075) 0:01:45.892 ******* 2022-01-11T13:53:29.352 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2022-01-11T13:53:29.353 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:29.376 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.044) 0:01:45.936 ******* 2022-01-11T13:53:29.405 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2022-01-11T13:53:29.430 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.054) 0:01:45.990 ******* 2022-01-11T13:53:29.448 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2022-01-11T13:53:29.450 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2022-01-11T13:53:29.475 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.044) 0:01:46.035 ******* 2022-01-11T13:53:29.503 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2022-01-11T13:53:29.505 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:29.527 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.052) 0:01:46.087 ******* 2022-01-11T13:53:29.544 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2022-01-11T13:53:29.547 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:29.577 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.050) 0:01:46.137 ******* 2022-01-11T13:53:29.600 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2022-01-11T13:53:29.602 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:29.627 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.049) 0:01:46.187 ******* 2022-01-11T13:53:29.657 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2022-01-11T13:53:29.658 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2022-01-11T13:53:29.692 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.064) 0:01:46.252 ******* 2022-01-11T13:53:29.713 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2022-01-11T13:53:29.716 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:29.749 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.056) 0:01:46.308 ******* 2022-01-11T13:53:29.767 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2022-01-11T13:53:29.769 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:29.792 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.044) 0:01:46.352 ******* 2022-01-11T13:53:29.821 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2022-01-11T13:53:29.823 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:29.845 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.052) 0:01:46.405 ******* 2022-01-11T13:53:29.873 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2022-01-11T13:53:29.875 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:29.897 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.052) 0:01:46.457 ******* 2022-01-11T13:53:29.916 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2022-01-11T13:53:29.940 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.042) 0:01:46.500 ******* 2022-01-11T13:53:29.969 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2022-01-11T13:53:29.971 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:29.993 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:29 +0000 (0:00:00.053) 0:01:46.553 ******* 2022-01-11T13:53:30.032 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2022-01-11T13:53:30.033 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.057 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.063) 0:01:46.617 ******* 2022-01-11T13:53:30.074 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.098 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.040) 0:01:46.658 ******* 2022-01-11T13:53:30.125 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2022-01-11T13:53:30.127 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.150 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.051) 0:01:46.710 ******* 2022-01-11T13:53:30.167 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2022-01-11T13:53:30.168 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2022-01-11T13:53:30.191 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.040) 0:01:46.750 ******* 2022-01-11T13:53:30.218 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2022-01-11T13:53:30.222 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.242 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.051) 0:01:46.802 ******* 2022-01-11T13:53:30.270 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2022-01-11T13:53:30.272 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.294 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.051) 0:01:46.854 ******* 2022-01-11T13:53:30.322 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2022-01-11T13:53:30.324 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.346 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.051) 0:01:46.906 ******* 2022-01-11T13:53:30.375 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2022-01-11T13:53:30.376 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.398 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.052) 0:01:46.958 ******* 2022-01-11T13:53:30.416 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2022-01-11T13:53:30.418 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.440 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.042) 0:01:47.000 ******* 2022-01-11T13:53:30.469 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2022-01-11T13:53:30.471 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.493 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.052) 0:01:47.053 ******* 2022-01-11T13:53:30.511 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.539 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.042) 0:01:47.096 ******* 2022-01-11T13:53:30.575 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.603 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.066) 0:01:47.163 ******* 2022-01-11T13:53:30.621 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2022-01-11T13:53:30.623 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2022-01-11T13:53:30.646 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.043) 0:01:47.206 ******* 2022-01-11T13:53:30.674 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2022-01-11T13:53:30.676 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.700 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.053) 0:01:47.260 ******* 2022-01-11T13:53:30.719 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2022-01-11T13:53:30.721 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.745 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.044) 0:01:47.305 ******* 2022-01-11T13:53:30.774 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.798 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.053) 0:01:47.358 ******* 2022-01-11T13:53:30.818 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:30.842 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.044) 0:01:47.403 ******* 2022-01-11T13:53:30.882 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2022-01-11T13:53:30.884 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2022-01-11T13:53:30.908 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.065) 0:01:47.468 ******* 2022-01-11T13:53:30.979 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2022-01-11T13:53:30.980 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:31.001 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:30 +0000 (0:00:00.092) 0:01:47.561 ******* 2022-01-11T13:53:31.081 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2022-01-11T13:53:31.083 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:31.104 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:31 +0000 (0:00:00.103) 0:01:47.664 ******* 2022-01-11T13:53:31.122 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2022-01-11T13:53:31.124 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2022-01-11T13:53:31.147 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:31 +0000 (0:00:00.042) 0:01:47.707 ******* 2022-01-11T13:53:31.373 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2022-01-11T13:53:31.396 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:31 +0000 (0:00:00.249) 0:01:47.956 ******* 2022-01-11T13:53:31.929 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2022-01-11T13:53:31.932 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} ok: [smithi172.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2022-01-11T13:53:32.372 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2022-01-11T13:53:32.375 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2022-01-11T13:53:32.403 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:32 +0000 (0:00:01.006) 0:01:48.963 ******* 2022-01-11T13:53:32.427 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:32 +0000 (0:00:00.024) 0:01:48.987 ******* 2022-01-11T13:53:32.933 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2022-01-11T13:53:32.935 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2022-01-11T13:53:32.948 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2022-01-11T13:53:33.380 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2022-01-11T13:53:33.406 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:33 +0000 (0:00:00.978) 0:01:49.966 ******* 2022-01-11T13:53:33.414 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2022-01-11T13:53:33.441 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:33 +0000 (0:00:00.034) 0:01:50.001 ******* 2022-01-11T13:53:33.500 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2022-01-11T13:53:33.503 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:33.523 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:33 +0000 (0:00:00.081) 0:01:50.083 ******* 2022-01-11T13:53:33.530 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:33.554 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:33 +0000 (0:00:00.031) 0:01:50.114 ******* 2022-01-11T13:53:33.595 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:33.619 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:33 +0000 (0:00:00.064) 0:01:50.179 ******* 2022-01-11T13:53:33.636 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:33.662 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:33 +0000 (0:00:00.041) 0:01:50.221 ******* 2022-01-11T13:53:33.700 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:33.726 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:33 +0000 (0:00:00.063) 0:01:50.284 ******* 2022-01-11T13:53:33.742 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:33.767 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:33 +0000 (0:00:00.043) 0:01:50.327 ******* 2022-01-11T13:53:33.795 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2022-01-11T13:53:33.797 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:33.820 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:33 +0000 (0:00:00.052) 0:01:50.380 ******* 2022-01-11T13:53:33.849 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:33.874 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:33 +0000 (0:00:00.054) 0:01:50.434 ******* 2022-01-11T13:53:33.903 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2022-01-11T13:53:33.906 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:33.915 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:33.940 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:33 +0000 (0:00:00.065) 0:01:50.499 ******* 2022-01-11T13:53:33.958 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:33.960 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:33.970 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:33.993 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:33 +0000 (0:00:00.053) 0:01:50.553 ******* 2022-01-11T13:53:34.010 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2022-01-11T13:53:34.012 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:34.035 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:34 +0000 (0:00:00.042) 0:01:50.595 ******* 2022-01-11T13:53:34.063 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:34.087 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:34 +0000 (0:00:00.051) 0:01:50.647 ******* 2022-01-11T13:53:34.105 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2022-01-11T13:53:34.107 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:34.130 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:34 +0000 (0:00:00.042) 0:01:50.690 ******* 2022-01-11T13:53:34.180 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:34.182 INFO:teuthology.task.ansible.out: 2022-01-11T13:53:34.208 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:34 +0000 (0:00:00.078) 0:01:50.768 ******* 2022-01-11T13:53:34.215 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2022-01-11T13:53:34.217 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:34.240 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:34 +0000 (0:00:00.032) 0:01:50.801 ******* 2022-01-11T13:53:34.279 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:34.303 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:34 +0000 (0:00:00.062) 0:01:50.863 ******* 2022-01-11T13:53:34.310 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2022-01-11T13:53:34.312 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:34.334 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:34 +0000 (0:00:00.031) 0:01:50.895 ******* 2022-01-11T13:53:34.372 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:34.398 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:34 +0000 (0:00:00.063) 0:01:50.958 ******* 2022-01-11T13:53:34.404 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2022-01-11T13:53:34.406 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:34.430 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:34 +0000 (0:00:00.031) 0:01:50.990 ******* 2022-01-11T13:53:34.458 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:34.482 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:34 +0000 (0:00:00.052) 0:01:51.042 ******* 2022-01-11T13:53:35.518 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2022-01-11T13:53:35.520 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.837567", "end": "2022-01-11 13:53:35.500340", "rc": 0, "start": "2022-01-11 13:53:34.662773", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:53:35.542 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:35 +0000 (0:00:01.059) 0:01:52.102 ******* 2022-01-11T13:53:36.528 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.897631", "end": "2022-01-11 13:53:36.501448", "rc": 0, "start": "2022-01-11 13:53:34.603817", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:53:36.552 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:36 +0000 (0:00:01.009) 0:01:53.112 ******* 2022-01-11T13:53:39.567 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2022-01-11T13:53:39.569 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:53:39.590 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:39 +0000 (0:00:03.038) 0:01:56.150 ******* 2022-01-11T13:53:39.659 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2022-01-11T13:53:39.661 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:39.682 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:39 +0000 (0:00:00.092) 0:01:56.242 ******* 2022-01-11T13:53:39.751 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:39.774 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:39 +0000 (0:00:00.091) 0:01:56.334 ******* 2022-01-11T13:53:39.866 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:39.888 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:39 +0000 (0:00:00.114) 0:01:56.448 ******* 2022-01-11T13:53:39.957 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2022-01-11T13:53:39.959 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:39.980 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:39 +0000 (0:00:00.092) 0:01:56.540 ******* 2022-01-11T13:53:40.260 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2022-01-11T13:53:40.262 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:53:40.290 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:40 +0000 (0:00:00.309) 0:01:56.850 ******* 2022-01-11T13:53:40.360 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:40.382 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:40 +0000 (0:00:00.092) 0:01:56.942 ******* 2022-01-11T13:53:40.451 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2022-01-11T13:53:40.453 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:40.473 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:40 +0000 (0:00:00.091) 0:01:57.033 ******* 2022-01-11T13:53:40.480 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2022-01-11T13:53:40.481 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2022-01-11T13:53:40.505 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:40 +0000 (0:00:00.031) 0:01:57.065 ******* 2022-01-11T13:53:40.544 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2022-01-11T13:53:40.546 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:40.568 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:40 +0000 (0:00:00.062) 0:01:57.128 ******* 2022-01-11T13:53:40.637 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2022-01-11T13:53:40.639 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:40.661 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:40 +0000 (0:00:00.092) 0:01:57.221 ******* 2022-01-11T13:53:41.162 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2022-01-11T13:53:41.164 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2022-01-11T13:53:41.185 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:41 +0000 (0:00:00.524) 0:01:57.745 ******* 2022-01-11T13:53:42.062 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2022-01-11T13:53:42.064 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:53:42.094 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:42 +0000 (0:00:00.908) 0:01:58.654 ******* 2022-01-11T13:53:42.647 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2022-01-11T13:53:42.649 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2022-01-11T13:53:42.746 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:53:42.769 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:42 +0000 (0:00:00.673) 0:01:59.328 ******* 2022-01-11T13:53:43.084 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2022-01-11T13:53:43.086 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2022-01-11T13:53:43.294 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2022-01-11T13:53:43.529 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2022-01-11T13:53:43.554 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:43 +0000 (0:00:00.785) 0:02:00.114 ******* 2022-01-11T13:53:43.731 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2022-01-11T13:53:43.754 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2022-01-11T13:53:43.777 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:43 +0000 (0:00:00.223) 0:02:00.337 ******* 2022-01-11T13:53:44.162 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2022-01-11T13:53:44.164 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2022-01-11T13:53:44.188 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:44 +0000 (0:00:00.411) 0:02:00.748 ******* 2022-01-11T13:53:44.268 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2022-01-11T13:53:44.270 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2022-01-11T13:53:44.384 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2022-01-11T13:53:44.386 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2022-01-11T13:53:44.406 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:44 +0000 (0:00:00.218) 0:02:00.966 ******* 2022-01-11T13:53:44.696 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2022-01-11T13:53:44.700 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2022-01-11T13:53:44.928 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2022-01-11T13:53:45.109 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2022-01-11T13:53:45.173 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:45 +0000 (0:00:00.766) 0:02:01.733 ******* 2022-01-11T13:53:45.213 INFO:teuthology.task.ansible.out: TASK [common : Include dnf_system vars] **************************************** 2022-01-11T13:53:45.215 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/dnf_systems.yml"], "changed": false} 2022-01-11T13:53:45.236 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:45 +0000 (0:00:00.062) 0:02:01.796 ******* 2022-01-11T13:53:45.338 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2022-01-11T13:53:45.341 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2022-01-11T13:53:45.424 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* changed: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2022-01-11T13:53:45.447 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:45 +0000 (0:00:00.211) 0:02:02.008 ******* 2022-01-11T13:53:45.486 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi172.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:45.497 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:45.508 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:45.520 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:45.543 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:45 +0000 (0:00:00.095) 0:02:02.103 ******* 2022-01-11T13:53:45.762 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2022-01-11T13:53:45.764 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2022-01-11T13:53:45.788 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:45 +0000 (0:00:00.244) 0:02:02.348 ******* 2022-01-11T13:53:45.826 INFO:teuthology.task.ansible.out: TASK [common : Include dnf_system vars] **************************************** ok: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/dnf_systems.yml"], "changed": false} 2022-01-11T13:53:45.848 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:45 +0000 (0:00:00.060) 0:02:02.408 ******* 2022-01-11T13:53:46.043 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2022-01-11T13:53:46.045 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2022-01-11T13:53:46.065 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:46 +0000 (0:00:00.216) 0:02:02.625 ******* 2022-01-11T13:53:46.113 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2022-01-11T13:53:46.115 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:46.125 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:46.136 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:46.137 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:46.158 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:46 +0000 (0:00:00.093) 0:02:02.719 ******* 2022-01-11T13:53:47.049 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:53:47.071 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:47 +0000 (0:00:00.912) 0:02:03.631 ******* 2022-01-11T13:53:47.100 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (non-RHEL/CentOS)] ******** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:47.121 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:47 +0000 (0:00:00.050) 0:02:03.681 ******* 2022-01-11T13:53:47.580 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2022-01-11T13:53:47.602 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:47 +0000 (0:00:00.480) 0:02:04.161 ******* 2022-01-11T13:53:47.649 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2022-01-11T13:53:47.651 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:53:47.674 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:47 +0000 (0:00:00.071) 0:02:04.233 ******* 2022-01-11T13:53:47.700 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (non-RHEL/CentOS)] ******** 2022-01-11T13:53:47.703 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:47.723 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:47 +0000 (0:00:00.049) 0:02:04.283 ******* 2022-01-11T13:53:48.089 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2022-01-11T13:53:48.091 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2022-01-11T13:53:48.120 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:48 +0000 (0:00:00.396) 0:02:04.680 ******* 2022-01-11T13:53:48.159 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2022-01-11T13:53:48.161 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:48.184 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:48 +0000 (0:00:00.063) 0:02:04.744 ******* 2022-01-11T13:53:48.202 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2022-01-11T13:53:48.204 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2022-01-11T13:53:48.229 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:48 +0000 (0:00:00.044) 0:02:04.789 ******* 2022-01-11T13:53:48.380 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2022-01-11T13:53:48.405 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:48 +0000 (0:00:00.173) 0:02:04.963 ******* 2022-01-11T13:53:48.640 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [smithi172.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2022-01-11T13:53:48.665 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:48 +0000 (0:00:00.261) 0:02:05.224 ******* 2022-01-11T13:53:48.703 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2022-01-11T13:53:48.706 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:48.727 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:48 +0000 (0:00:00.063) 0:02:05.287 ******* 2022-01-11T13:53:48.756 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2022-01-11T13:53:48.784 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:48 +0000 (0:00:00.056) 0:02:05.344 ******* 2022-01-11T13:53:48.825 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2022-01-11T13:53:48.827 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:48.837 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2022-01-11T13:53:48.861 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:48 +0000 (0:00:00.076) 0:02:05.421 ******* 2022-01-11T13:53:49.060 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi161.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2022-01-11T13:53:49.083 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:49 +0000 (0:00:00.221) 0:02:05.642 ******* 2022-01-11T13:53:49.193 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2022-01-11T13:53:49.195 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2022-01-11T13:53:49.218 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:49 +0000 (0:00:00.136) 0:02:05.778 ******* 2022-01-11T13:53:49.319 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2022-01-11T13:53:49.321 INFO:teuthology.task.ansible.out:fatal: [smithi161.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2022-01-11T13:53:49.342 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:49 +0000 (0:00:00.124) 0:02:05.902 ******* 2022-01-11T13:53:49.383 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:53:49.409 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:49 +0000 (0:00:00.066) 0:02:05.969 ******* 2022-01-11T13:53:49.851 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1248", "ExecMainStartTimestamp": "Tue 2022-01-11 13:50:46 UTC", "ExecMainStartTimestampMonotonic": "19856140", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Tue 2022-01-11 13:50:46 UTC] ; stop_time=[n/a] ; pid=1248 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "984", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "1844674407370955 2022-01-11T13:53:49.853 INFO:teuthology.task.ansible.out:1615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "1248", "MemoryAccounting": "yes", "MemoryCurrent": "675840", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "on-abort", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no 2022-01-11T13:53:49.855 INFO:teuthology.task.ansible.out:", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Type": "simple", "UID": "988", "UMask": "0022", "WatchdogTimestamp": "Tue 2022-01-11 13:50:46 UTC", "WatchdogTimestampMonotonic": "19856202", "WatchdogUSec": "0"}} 2022-01-11T13:53:49.880 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:49 +0000 (0:00:00.470) 0:02:06.440 ******* 2022-01-11T13:53:49.899 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2022-01-11T13:53:49.901 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2022-01-11T13:53:49.926 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:49 +0000 (0:00:00.045) 0:02:06.486 ******* 2022-01-11T13:53:50.065 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001797", "end": "2022-01-11 13:53:50.049987", "rc": 0, "start": "2022-01-11 13:53:50.048190", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2022-01-11T13:53:50.088 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:50 +0000 (0:00:00.162) 0:02:06.648 ******* 2022-01-11T13:53:50.244 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2022-01-11T13:53:50.246 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1281", "ExecMainStartTimestamp": "Tue 2022-01-11 13:50:49 UTC", "ExecMainStartTimestampMonotonic": "20049989", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Tue 2022-01-11 13:50:49 UTC] ; stop_time=[n/a] ; pid=1281 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "984", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "L 2022-01-11T13:53:50.249 INFO:teuthology.task.ansible.out:imitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "1281", "MemoryAccounting": "yes", "MemoryCurrent": "663552", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "on-abort", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203592", "TimeoutStar 2022-01-11T13:53:50.250 INFO:teuthology.task.ansible.out:tUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Type": "simple", "UID": "988", "UMask": "0022", "WatchdogTimestamp": "Tue 2022-01-11 13:50:49 UTC", "WatchdogTimestampMonotonic": "20050161", "WatchdogUSec": "0"}} 2022-01-11T13:53:50.270 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:50 +0000 (0:00:00.182) 0:02:06.830 ******* 2022-01-11T13:53:50.457 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2022-01-11T13:53:50.459 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001878", "end": "2022-01-11 13:53:50.439373", "rc": 0, "start": "2022-01-11 13:53:50.437495", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2022-01-11T13:53:50.479 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:50 +0000 (0:00:00.208) 0:02:07.039 ******* 2022-01-11T13:53:51.634 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2022-01-11T13:53:51.636 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:53:51.658 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:51 +0000 (0:00:01.178) 0:02:08.218 ******* 2022-01-11T13:53:51.969 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:53:51.995 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:53:51 +0000 (0:00:00.337) 0:02:08.555 ******* 2022-01-11T13:54:30.052 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2022-01-11T13:54:30.054 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-80.el8_5.2.noarch", "Installed: selinux-policy-3.14.3-80.el8_5.2.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2022-01-11T13:54:30.078 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:54:30 +0000 (0:00:38.083) 0:02:46.638 ******* 2022-01-11T13:54:30.341 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-80.el8_5.2.noarch", "Installed: selinux-policy-3.14.3-80.el8_5.2.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2022-01-11T13:54:30.364 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:54:30 +0000 (0:00:00.286) 0:02:46.924 ******* 2022-01-11T13:54:30.786 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2022-01-11T13:54:30.788 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2022-01-11T13:54:30.810 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:54:30 +0000 (0:00:00.445) 0:02:47.370 ******* 2022-01-11T13:54:30.817 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2022-01-11T13:54:30.841 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:54:30 +0000 (0:00:00.030) 0:02:47.401 ******* 2022-01-11T13:54:46.036 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2022-01-11T13:54:46.038 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:14.998834", "end": "2022-01-11 13:54:46.016257", "failed_when_result": false, "rc": 0, "start": "2022-01-11 13:54:31.017423", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.038745", "end": "2022-01-11 13:54:46.017854", "failed_when_result": false, "rc": 0, "start": "2022-01-11 13:54:30.979109", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2022-01-11T13:54:46.058 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:54:46 +0000 (0:00:15.217) 0:03:02.618 ******* 2022-01-11T13:54:46.078 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:54:46 +0000 (0:00:00.019) 0:03:02.638 ******* 2022-01-11T13:54:46.524 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2022-01-11T13:54:46.526 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2022-01-11T13:54:46.553 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:54:46 +0000 (0:00:00.475) 0:03:03.113 ******* 2022-01-11T13:54:46.564 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2022-01-11T13:54:46.593 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:54:46 +0000 (0:00:00.039) 0:03:03.153 ******* 2022-01-11T13:54:46.755 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2022-01-11T13:54:46.757 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.012859", "end": "2022-01-11 13:54:46.733907", "rc": 0, "start": "2022-01-11 13:54:46.721048", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:54:46.779 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:54:46 +0000 (0:00:00.186) 0:03:03.340 ******* 2022-01-11T13:54:46.798 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.010452", "end": "2022-01-11 13:54:46.774476", "rc": 0, "start": "2022-01-11 13:54:46.764024", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:54:46.822 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:54:46 +0000 (0:00:00.042) 0:03:03.383 ******* 2022-01-11T13:54:46.982 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2022-01-11T13:54:46.984 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.015455", "end": "2022-01-11 13:54:46.962737", "rc": 0, "start": "2022-01-11 13:54:46.947282", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:54:47.008 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:54:47 +0000 (0:00:00.185) 0:03:03.568 ******* 2022-01-11T13:54:47.027 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.013045", "end": "2022-01-11 13:54:47.013046", "rc": 0, "start": "2022-01-11 13:54:47.000001", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:54:47.051 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:54:47 +0000 (0:00:00.043) 0:03:03.612 ******* 2022-01-11T13:55:02.220 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2022-01-11T13:55:02.222 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:14.975019", "end": "2022-01-11 13:55:02.198967", "rc": 0, "start": "2022-01-11 13:54:47.223948", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:55:02.242 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:02 +0000 (0:00:15.190) 0:03:18.802 ******* 2022-01-11T13:55:02.354 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.157327", "end": "2022-01-11 13:55:02.335744", "rc": 0, "start": "2022-01-11 13:54:47.178417", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:55:02.377 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:02 +0000 (0:00:00.135) 0:03:18.938 ******* 2022-01-11T13:55:02.448 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2022-01-11T13:55:02.450 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2022-01-11T13:55:02.472 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:02 +0000 (0:00:00.094) 0:03:19.032 ******* 2022-01-11T13:55:02.510 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2022-01-11T13:55:02.512 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2022-01-11T13:55:02.534 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:02 +0000 (0:00:00.061) 0:03:19.094 ******* 2022-01-11T13:55:02.572 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* ok: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2022-01-11T13:55:02.573 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2022-01-11T13:55:02.573 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2022-01-11T13:55:02.596 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:02 +0000 (0:00:00.061) 0:03:19.155 ******* 2022-01-11T13:55:02.614 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:02 +0000 (0:00:00.019) 0:03:19.174 ******* 2022-01-11T13:55:02.622 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2022-01-11T13:55:02.624 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2022-01-11T13:55:02.647 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:02 +0000 (0:00:00.032) 0:03:19.207 ******* 2022-01-11T13:55:02.655 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2022-01-11T13:55:02.656 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2022-01-11T13:55:02.658 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2022-01-11T13:55:02.682 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:02 +0000 (0:00:00.034) 0:03:19.242 ******* 2022-01-11T13:55:02.689 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* ok: [smithi161.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2022-01-11T13:55:02.715 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:02 +0000 (0:00:00.032) 0:03:19.275 ******* 2022-01-11T13:55:02.723 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2022-01-11T13:55:02.724 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2022-01-11T13:55:02.749 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:02 +0000 (0:00:00.034) 0:03:19.309 ******* 2022-01-11T13:55:02.758 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2022-01-11T13:55:02.759 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2022-01-11T13:55:02.761 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2022-01-11T13:55:02.785 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:02 +0000 (0:00:00.035) 0:03:19.345 ******* 2022-01-11T13:55:02.825 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2022-01-11T13:55:02.827 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2022-01-11T13:55:02.849 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:02 +0000 (0:00:00.063) 0:03:19.409 ******* 2022-01-11T13:55:02.940 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2022-01-11T13:55:02.942 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2022-01-11T13:55:02.973 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:02 +0000 (0:00:00.124) 0:03:19.533 ******* 2022-01-11T13:55:03.033 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2022-01-11T13:55:03.057 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:03 +0000 (0:00:00.083) 0:03:19.617 ******* 2022-01-11T13:55:03.168 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2022-01-11T13:55:03.169 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "name": "ubuntu", "state": "present", "system": false} 2022-01-11T13:55:03.192 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:03 +0000 (0:00:00.134) 0:03:19.752 ******* 2022-01-11T13:55:03.219 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2022-01-11T13:55:03.221 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:03.242 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:03 +0000 (0:00:00.049) 0:03:19.802 ******* 2022-01-11T13:55:03.249 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2022-01-11T13:55:03.250 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "name": "ubuntu", "state": "present", "system": false} 2022-01-11T13:55:03.272 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:03 +0000 (0:00:00.030) 0:03:19.833 ******* 2022-01-11T13:55:03.279 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2022-01-11T13:55:03.280 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:03.304 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:03 +0000 (0:00:00.031) 0:03:19.864 ******* 2022-01-11T13:55:03.311 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2022-01-11T13:55:03.313 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:03.337 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:03 +0000 (0:00:00.033) 0:03:19.897 ******* 2022-01-11T13:55:03.365 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2022-01-11T13:55:03.367 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:03.389 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:03 +0000 (0:00:00.051) 0:03:19.949 ******* 2022-01-11T13:55:03.520 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2022-01-11T13:55:03.522 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2022-01-11T13:55:03.546 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:03 +0000 (0:00:00.156) 0:03:20.106 ******* 2022-01-11T13:55:03.606 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2022-01-11T13:55:03.629 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:03 +0000 (0:00:00.083) 0:03:20.190 ******* 2022-01-11T13:55:04.087 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2022-01-11T13:55:04.109 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:04 +0000 (0:00:00.479) 0:03:20.669 ******* 2022-01-11T13:55:04.157 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2022-01-11T13:55:04.159 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi172"}, "changed": false} 2022-01-11T13:55:04.180 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:04 +0000 (0:00:00.071) 0:03:20.740 ******* 2022-01-11T13:55:04.187 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2022-01-11T13:55:04.189 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2022-01-11T13:55:04.212 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:04 +0000 (0:00:00.031) 0:03:20.772 ******* 2022-01-11T13:55:04.251 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2022-01-11T13:55:04.252 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi161"}, "changed": false} 2022-01-11T13:55:04.273 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:04 +0000 (0:00:00.061) 0:03:20.833 ******* 2022-01-11T13:55:05.086 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2022-01-11T13:55:05.161 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi172.front.sepia.ceph.com", "ansible_hostname": "smithi172", "ansible_nodename": "smithi172"}, "changed": true, "name": "smithi172"} Tuesday 11 January 2022 13:55:05 +0000 (0:00:00.863) 0:03:21.696 ******* changed: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi161.front.sepia.ceph.com", "ansible_hostname": "smithi161", "ansible_nodename": "smithi161"}, "changed": true, "name": "smithi161"} 2022-01-11T13:55:05.174 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:05 +0000 (0:00:00.037) 0:03:21.734 ******* 2022-01-11T13:55:05.598 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2022-01-11T13:55:05.600 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2022-01-11T13:55:05.622 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:05 +0000 (0:00:00.447) 0:03:22.182 ******* 2022-01-11T13:55:05.640 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2022-01-11T13:55:05.664 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:05 +0000 (0:00:00.041) 0:03:22.224 ******* 2022-01-11T13:55:06.076 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2022-01-11T13:55:06.078 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2022-01-11T13:55:06.101 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:06 +0000 (0:00:00.436) 0:03:22.660 ******* 2022-01-11T13:55:06.139 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2022-01-11T13:55:06.163 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:06 +0000 (0:00:00.062) 0:03:22.723 ******* 2022-01-11T13:55:06.620 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2022-01-11T13:55:06.622 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2022-01-11T13:55:06.646 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:06 +0000 (0:00:00.482) 0:03:23.206 ******* 2022-01-11T13:55:06.708 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2022-01-11T13:55:06.732 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:06 +0000 (0:00:00.086) 0:03:23.292 ******* 2022-01-11T13:55:07.112 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2022-01-11T13:55:07.137 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:07 +0000 (0:00:00.404) 0:03:23.697 ******* 2022-01-11T13:55:07.206 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2022-01-11T13:55:07.230 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:07 +0000 (0:00:00.093) 0:03:23.790 ******* 2022-01-11T13:55:07.332 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2022-01-11T13:55:07.334 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2022-01-11T13:55:07.357 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:07 +0000 (0:00:00.126) 0:03:23.917 ******* 2022-01-11T13:55:07.439 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2022-01-11T13:55:07.465 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:07 +0000 (0:00:00.107) 0:03:24.025 ******* 2022-01-11T13:55:07.558 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2022-01-11T13:55:07.560 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2022-01-11T13:55:07.588 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:07 +0000 (0:00:00.122) 0:03:24.148 ******* 2022-01-11T13:55:07.673 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2022-01-11T13:55:07.703 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:07 +0000 (0:00:00.115) 0:03:24.263 ******* 2022-01-11T13:55:07.825 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2022-01-11T13:55:07.827 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1254, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2022-01-11T13:55:07.850 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:07 +0000 (0:00:00.146) 0:03:24.410 ******* 2022-01-11T13:55:07.931 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1254, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2022-01-11T13:55:07.955 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:07 +0000 (0:00:00.104) 0:03:24.515 ******* 2022-01-11T13:55:08.318 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2022-01-11T13:55:08.319 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2022-01-11T13:55:08.342 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:08 +0000 (0:00:00.387) 0:03:24.902 ******* 2022-01-11T13:55:08.424 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2022-01-11T13:55:08.448 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:08 +0000 (0:00:00.105) 0:03:25.008 ******* 2022-01-11T13:55:08.809 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2022-01-11T13:55:08.814 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2022-01-11T13:55:08.832 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:08 +0000 (0:00:00.384) 0:03:25.392 ******* 2022-01-11T13:55:08.913 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2022-01-11T13:55:08.943 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:08 +0000 (0:00:00.110) 0:03:25.503 ******* 2022-01-11T13:55:11.425 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2022-01-11T13:55:11.427 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.288959", "end": "2022-01-11 13:55:11.400980", "rc": 0, "start": "2022-01-11 13:55:09.112021", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:55:11.450 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:11 +0000 (0:00:02.507) 0:03:28.011 ******* 2022-01-11T13:55:11.484 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.459895", "end": "2022-01-11 13:55:11.460807", "rc": 0, "start": "2022-01-11 13:55:09.000912", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Clean up Stream distro-provided repos] ************************ skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:11.512 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:11 +0000 (0:00:00.061) 0:03:28.072 ******* 2022-01-11T13:55:11.538 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:11 +0000 (0:00:00.025) 0:03:28.097 ******* 2022-01-11T13:55:11.557 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:11.582 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:11 +0000 (0:00:00.044) 0:03:28.141 ******* 2022-01-11T13:55:11.589 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2022-01-11T13:55:11.591 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:11.615 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:11 +0000 (0:00:00.033) 0:03:28.175 ******* 2022-01-11T13:55:11.632 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2022-01-11T13:55:11.634 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml", "skip_reason": "Conditional result was False"} 2022-01-11T13:55:11.656 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:11 +0000 (0:00:00.041) 0:03:28.216 ******* 2022-01-11T13:55:11.664 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2022-01-11T13:55:11.666 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:11.690 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:11 +0000 (0:00:00.033) 0:03:28.250 ******* 2022-01-11T13:55:11.740 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2022-01-11T13:55:11.741 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml", "skip_reason": "Conditional result was False"} 2022-01-11T13:55:11.764 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:11 +0000 (0:00:00.074) 0:03:28.324 ******* 2022-01-11T13:55:12.155 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2022-01-11T13:55:12.157 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2022-01-11T13:55:12.262 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2022-01-11T13:55:12.586 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2022-01-11T13:55:12.690 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2022-01-11T13:55:13.032 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2022-01-11T13:55:13.127 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2022-01-11T13:55:13.470 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "20107eac0ba9a84b318fa5f29b02a387051bff98", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2022-01-11T13:55:13.495 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:13 +0000 (0:00:01.730) 0:03:30.055 ******* 2022-01-11T13:55:13.556 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "20107eac0ba9a84b318fa5f29b02a387051bff98", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2022-01-11T13:55:13.584 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:13 +0000 (0:00:00.089) 0:03:30.144 ******* 2022-01-11T13:55:13.965 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2022-01-11T13:55:13.967 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2022-01-11T13:55:14.061 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2022-01-11T13:55:14.449 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'apache-arrow', 'value': {'name': 'Apache Arrow', 'baseurl': 'https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch', 'enabled': 1, 'gpgcheck': 1}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "86b924db6e383ddad921a207c09a708e6cca6a67", "dest": "/etc/yum.repos.d/apache-arrow.repo", "gid": 0, "group": "root", "item": {"key": "apache-arrow", "value": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}}, "md5sum": "183d0889f7213a1e38c7f48965d7549e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 215, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1641909314.0019512-17803-108260825845786/source", "state": "file", "uid": 0} 2022-01-11T13:55:14.475 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:14 +0000 (0:00:00.890) 0:03:31.035 ******* 2022-01-11T13:55:14.544 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'apache-arrow', 'value': {'name': 'Apache Arrow', 'baseurl': 'https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch', 'enabled': 1, 'gpgcheck': 1}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "86b924db6e383ddad921a207c09a708e6cca6a67", "dest": "/etc/yum.repos.d/apache-arrow.repo", "gid": 0, "group": "root", "item": {"key": "apache-arrow", "value": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}}, "md5sum": "183d0889f7213a1e38c7f48965d7549e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 215, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1641909314.0899017-17822-132720015189824/source", "state": "file", "uid": 0} 2022-01-11T13:55:14.570 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:14 +0000 (0:00:00.095) 0:03:31.130 ******* 2022-01-11T13:55:14.948 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2022-01-11T13:55:14.950 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2022-01-11T13:55:15.048 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2022-01-11T13:55:15.387 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2022-01-11T13:55:15.484 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2022-01-11T13:55:15.855 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2022-01-11T13:55:15.966 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2022-01-11T13:55:16.296 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "state": "file", "uid": 0} 2022-01-11T13:55:16.323 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:16 +0000 (0:00:01.752) 0:03:32.883 ******* 2022-01-11T13:55:16.404 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "state": "file", "uid": 0} 2022-01-11T13:55:16.433 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:16 +0000 (0:00:00.110) 0:03:32.994 ******* 2022-01-11T13:55:16.809 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2022-01-11T13:55:16.810 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.279309", "end": "2022-01-11 13:55:16.782426", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2022-01-11 13:55:16.503117", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2022-01-11T13:55:16.811 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'comma 2022-01-11T13:55:16.812 INFO:teuthology.task.ansible.out:nd_warnings=False' in ansible.cfg to get rid of this message. 2022-01-11T13:55:16.835 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:16 +0000 (0:00:00.401) 0:03:33.395 ******* 2022-01-11T13:55:16.916 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.279923", "end": "2022-01-11 13:55:16.894514", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2022-01-11 13:55:16.614591", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2022-01-11T13:55:16.943 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:16 +0000 (0:00:00.108) 0:03:33.503 ******* 2022-01-11T13:55:21.607 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2022-01-11T13:55:21.609 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:55:21.630 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:21 +0000 (0:00:04.686) 0:03:38.190 ******* 2022-01-11T13:55:21.638 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:55:21.664 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:21 +0000 (0:00:00.034) 0:03:38.224 ******* 2022-01-11T13:55:22.513 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2022-01-11T13:55:22.515 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/release.asc"} 2022-01-11T13:55:22.546 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/release.asc"} 2022-01-11T13:55:23.090 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/autobuild.asc"} 2022-01-11T13:55:23.102 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/autobuild.asc"} 2022-01-11T13:55:24.430 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item=https://dist.apache.org/repos/dist/dev/arrow/KEYS) => {"ansible_loop_var": "item", "changed": true, "item": "https://dist.apache.org/repos/dist/dev/arrow/KEYS"} 2022-01-11T13:55:24.452 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:24 +0000 (0:00:02.787) 0:03:41.012 ******* 2022-01-11T13:55:24.522 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=https://dist.apache.org/repos/dist/dev/arrow/KEYS) => {"ansible_loop_var": "item", "changed": true, "item": "https://dist.apache.org/repos/dist/dev/arrow/KEYS"} 2022-01-11T13:55:24.546 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:24 +0000 (0:00:00.094) 0:03:41.106 ******* 2022-01-11T13:55:24.840 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2022-01-11T13:55:24.842 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum module rather than running 'yum'. If you need to use command because yum is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.186812", "end": "2022-01-11 13:55:24.818941", "rc": 0, "start": "2022-01-11 13:55:24.632129", "stderr": "", "stderr_lines": [], "stdout": "78 files removed", "stdout_lines": ["78 files removed"]} 2022-01-11T13:55:24.866 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:24 +0000 (0:00:00.320) 0:03:41.426 ******* 2022-01-11T13:55:24.896 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2022-01-11T13:55:24.898 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2022-01-11T13:55:24.920 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:24 +0000 (0:00:00.053) 0:03:41.480 ******* 2022-01-11T13:55:24.940 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2022-01-11T13:55:24.942 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.187267", "end": "2022-01-11 13:55:24.917526", "rc": 0, "start": "2022-01-11 13:55:24.730259", "stderr": "", "stderr_lines": [], "stdout": "78 files removed", "stdout_lines": ["78 files removed"]} 2022-01-11T13:55:24.965 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:24 +0000 (0:00:00.045) 0:03:41.525 ******* 2022-01-11T13:55:24.977 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2022-01-11T13:55:24.978 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:25.000 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:24 +0000 (0:00:00.034) 0:03:41.560 ******* 2022-01-11T13:55:25.007 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2022-01-11T13:55:25.031 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:25 +0000 (0:00:00.031) 0:03:41.592 ******* 2022-01-11T13:55:25.037 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2022-01-11T13:55:25.039 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:25.061 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:25 +0000 (0:00:00.029) 0:03:41.621 ******* 2022-01-11T13:55:25.067 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2022-01-11T13:55:25.069 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:25.093 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:25 +0000 (0:00:00.031) 0:03:41.653 ******* 2022-01-11T13:55:25.131 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2022-01-11T13:55:25.132 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:25.153 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:25 +0000 (0:00:00.060) 0:03:41.713 ******* 2022-01-11T13:55:25.263 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2022-01-11T13:55:25.265 INFO:teuthology.task.ansible.out:fatal: [smithi161.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007496", "end": "2022-01-11 13:55:25.242065", "msg": "non-zero return code", "rc": 1, "start": "2022-01-11 13:55:25.234569", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2022-01-11T13:55:25.287 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:25 +0000 (0:00:00.134) 0:03:41.847 ******* 2022-01-11T13:55:25.325 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2022-01-11T13:55:25.327 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:25.348 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:25 +0000 (0:00:00.060) 0:03:41.908 ******* 2022-01-11T13:55:25.376 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2022-01-11T13:55:25.378 INFO:teuthology.task.ansible.out:fatal: [smithi172.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007618", "end": "2022-01-11 13:55:25.352938", "msg": "non-zero return code", "rc": 1, "start": "2022-01-11 13:55:25.345320", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2022-01-11T13:55:25.401 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:25 +0000 (0:00:00.053) 0:03:41.961 ******* 2022-01-11T13:55:25.439 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2022-01-11T13:55:25.441 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:25.462 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:25 +0000 (0:00:00.060) 0:03:42.022 ******* 2022-01-11T13:55:45.913 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2022-01-11T13:55:45.915 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64"]} 2022-01-11T13:55:45.936 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:45 +0000 (0:00:20.474) 0:04:02.496 ******* 2022-01-11T13:55:46.195 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64"]} 2022-01-11T13:55:46.218 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:46 +0000 (0:00:00.282) 0:04:02.778 ******* 2022-01-11T13:55:48.563 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2022-01-11T13:55:48.565 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2022-01-11T13:55:48.588 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:48 +0000 (0:00:02.369) 0:04:05.148 ******* 2022-01-11T13:55:48.838 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2022-01-11T13:55:48.862 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:48 +0000 (0:00:00.274) 0:04:05.422 ******* 2022-01-11T13:55:57.432 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:4.2.0-59.module_el8.5.0+1063+c9b9feff.1.x86_64", "Installed: dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-59.module_el8.5.0+1063+c9b9feff.1.x86_64", "Installed: python3-dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: qemu-kvm-common-15:4.2.0-59.module_el8.5.0+1063+c9b9feff.1.x86_64", "Installed: qemu-kvm-core-15:4.2.0-59.module_el8.5.0+1063+c9b9feff.1.x86_64", "Installed: yum-utils-4.0.21-3.el8.noarch", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Removed: yum-utils-4.0.18-4.el8.noarch", "Removed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: qemu-img-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64"]} 2022-01-11T13:55:57.456 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:57 +0000 (0:00:08.592) 0:04:14.015 ******* 2022-01-11T13:55:57.494 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2022-01-11T13:55:57.495 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:57.516 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:57 +0000 (0:00:00.060) 0:04:14.076 ******* 2022-01-11T13:55:57.702 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2022-01-11T13:55:57.704 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:4.2.0-59.module_el8.5.0+1063+c9b9feff.1.x86_64", "Installed: dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-59.module_el8.5.0+1063+c9b9feff.1.x86_64", "Installed: python3-dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: qemu-kvm-common-15:4.2.0-59.module_el8.5.0+1063+c9b9feff.1.x86_64", "Installed: qemu-kvm-core-15:4.2.0-59.module_el8.5.0+1063+c9b9feff.1.x86_64", "Installed: yum-utils-4.0.21-3.el8.noarch", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Removed: yum-utils-4.0.18-4.el8.noarch", "Removed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: qemu-img-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64"]} 2022-01-11T13:55:57.725 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:57 +0000 (0:00:00.209) 0:04:14.285 ******* 2022-01-11T13:55:57.763 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2022-01-11T13:55:57.765 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:55:57.785 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:57 +0000 (0:00:00.059) 0:04:14.345 ******* 2022-01-11T13:55:59.102 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2022-01-11T13:55:59.103 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:55:59.125 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:59 +0000 (0:00:01.340) 0:04:15.685 ******* 2022-01-11T13:55:59.364 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:55:59.388 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:55:59 +0000 (0:00:00.262) 0:04:15.948 ******* 2022-01-11T13:56:01.966 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2022-01-11T13:56:01.968 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: libgcrypt-1.8.5-6.el8.x86_64", "Removed: libgcrypt-1.8.5-4.el8.x86_64"]} 2022-01-11T13:56:01.989 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:01 +0000 (0:00:02.601) 0:04:18.549 ******* 2022-01-11T13:56:02.270 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: libgcrypt-1.8.5-6.el8.x86_64", "Removed: libgcrypt-1.8.5-4.el8.x86_64"]} 2022-01-11T13:56:02.296 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:02 +0000 (0:00:00.304) 0:04:18.854 ******* 2022-01-11T13:56:02.323 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2022-01-11T13:56:02.325 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveState": "inactive", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity 2022-01-11T13:56:02.327 INFO:teuthology.task.ansible.out:", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2022-01-11T13:56:02.348 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:02 +0000 (0:00:00.054) 0:04:18.908 ******* 2022-01-11T13:56:02.606 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveState": "inactive", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity 2022-01-11T13:56:02.608 INFO:teuthology.task.ansible.out:", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2022-01-11T13:56:02.629 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:02 +0000 (0:00:00.280) 0:04:19.189 ******* 2022-01-11T13:56:02.647 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2022-01-11T13:56:02.649 INFO:teuthology.task.ansible.out:fatal: [smithi161.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2022-01-11T13:56:02.672 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:02 +0000 (0:00:00.042) 0:04:19.232 ******* 2022-01-11T13:56:02.924 INFO:teuthology.task.ansible.out:fatal: [smithi172.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2022-01-11T13:56:02.947 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:02 +0000 (0:00:00.274) 0:04:19.507 ******* 2022-01-11T13:56:03.273 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2022-01-11T13:56:03.274 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2022-01-11T13:56:03.295 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.348) 0:04:19.855 ******* 2022-01-11T13:56:03.319 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.024) 0:04:19.879 ******* 2022-01-11T13:56:03.338 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2022-01-11T13:56:03.340 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.361 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.041) 0:04:19.921 ******* 2022-01-11T13:56:03.371 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.396 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.034) 0:04:19.956 ******* 2022-01-11T13:56:03.404 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2022-01-11T13:56:03.406 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.432 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.036) 0:04:19.992 ******* 2022-01-11T13:56:03.448 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.473 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.040) 0:04:20.033 ******* 2022-01-11T13:56:03.480 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2022-01-11T13:56:03.485 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.507 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.033) 0:04:20.066 ******* 2022-01-11T13:56:03.516 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.540 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.033) 0:04:20.100 ******* 2022-01-11T13:56:03.546 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2022-01-11T13:56:03.548 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.574 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.034) 0:04:20.134 ******* 2022-01-11T13:56:03.587 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.612 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.037) 0:04:20.172 ******* 2022-01-11T13:56:03.619 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2022-01-11T13:56:03.621 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.646 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.034) 0:04:20.206 ******* 2022-01-11T13:56:03.654 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.678 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.031) 0:04:20.238 ******* 2022-01-11T13:56:03.685 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2022-01-11T13:56:03.687 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.710 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.031) 0:04:20.270 ******* 2022-01-11T13:56:03.718 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.741 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.030) 0:04:20.301 ******* 2022-01-11T13:56:03.748 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2022-01-11T13:56:03.774 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.030) 0:04:20.331 ******* 2022-01-11T13:56:03.803 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.031) 0:04:20.363 ******* 2022-01-11T13:56:03.810 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2022-01-11T13:56:03.811 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.834 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.031) 0:04:20.394 ******* 2022-01-11T13:56:03.843 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.865 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.031) 0:04:20.426 ******* 2022-01-11T13:56:03.872 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.896 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.030) 0:04:20.456 ******* 2022-01-11T13:56:03.904 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.929 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.033) 0:04:20.489 ******* 2022-01-11T13:56:03.937 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2022-01-11T13:56:03.938 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:03.963 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.033) 0:04:20.523 ******* 2022-01-11T13:56:03.973 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.000 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:03 +0000 (0:00:00.036) 0:04:20.560 ******* 2022-01-11T13:56:04.007 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2022-01-11T13:56:04.009 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.035 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.035) 0:04:20.596 ******* 2022-01-11T13:56:04.043 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.067 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.031) 0:04:20.627 ******* 2022-01-11T13:56:04.074 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2022-01-11T13:56:04.098 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.030) 0:04:20.658 ******* 2022-01-11T13:56:04.129 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.030) 0:04:20.689 ******* 2022-01-11T13:56:04.136 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2022-01-11T13:56:04.137 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.162 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.032) 0:04:20.722 ******* 2022-01-11T13:56:04.169 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.192 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.030) 0:04:20.752 ******* 2022-01-11T13:56:04.198 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2022-01-11T13:56:04.200 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.226 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.033) 0:04:20.786 ******* 2022-01-11T13:56:04.234 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.265 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.039) 0:04:20.825 ******* 2022-01-11T13:56:04.274 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2022-01-11T13:56:04.276 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.299 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.033) 0:04:20.859 ******* 2022-01-11T13:56:04.306 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.330 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.030) 0:04:20.890 ******* 2022-01-11T13:56:04.336 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2022-01-11T13:56:04.337 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.360 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.030) 0:04:20.920 ******* 2022-01-11T13:56:04.367 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.391 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.031) 0:04:20.951 ******* 2022-01-11T13:56:04.399 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2022-01-11T13:56:04.402 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.423 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.031) 0:04:20.983 ******* 2022-01-11T13:56:04.431 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.455 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.032) 0:04:21.015 ******* 2022-01-11T13:56:04.462 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2022-01-11T13:56:04.464 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.487 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.032) 0:04:21.047 ******* 2022-01-11T13:56:04.495 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.519 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.031) 0:04:21.079 ******* 2022-01-11T13:56:04.526 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2022-01-11T13:56:04.528 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.550 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.030) 0:04:21.110 ******* 2022-01-11T13:56:04.557 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.581 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.031) 0:04:21.141 ******* 2022-01-11T13:56:04.588 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2022-01-11T13:56:04.590 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.612 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.030) 0:04:21.172 ******* 2022-01-11T13:56:04.618 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.642 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.030) 0:04:21.202 ******* 2022-01-11T13:56:04.649 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2022-01-11T13:56:04.651 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.676 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.032) 0:04:21.235 ******* 2022-01-11T13:56:04.684 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.708 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.032) 0:04:21.268 ******* 2022-01-11T13:56:04.714 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2022-01-11T13:56:04.716 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.739 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.031) 0:04:21.299 ******* 2022-01-11T13:56:04.747 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.771 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.031) 0:04:21.331 ******* 2022-01-11T13:56:04.777 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2022-01-11T13:56:04.779 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.806 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.035) 0:04:21.366 ******* 2022-01-11T13:56:04.814 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.838 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.031) 0:04:21.398 ******* 2022-01-11T13:56:04.844 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2022-01-11T13:56:04.847 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.870 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.031) 0:04:21.430 ******* 2022-01-11T13:56:04.879 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.903 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.032) 0:04:21.463 ******* 2022-01-11T13:56:04.909 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2022-01-11T13:56:04.911 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.936 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.033) 0:04:21.496 ******* 2022-01-11T13:56:04.945 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:04.969 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.033) 0:04:21.529 ******* 2022-01-11T13:56:04.976 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2022-01-11T13:56:04.977 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.001 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:04 +0000 (0:00:00.031) 0:04:21.561 ******* 2022-01-11T13:56:05.009 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.044 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.042) 0:04:21.603 ******* 2022-01-11T13:56:05.052 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2022-01-11T13:56:05.054 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.080 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.036) 0:04:21.640 ******* 2022-01-11T13:56:05.090 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.117 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.036) 0:04:21.677 ******* 2022-01-11T13:56:05.126 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2022-01-11T13:56:05.128 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.150 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.033) 0:04:21.710 ******* 2022-01-11T13:56:05.157 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.181 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.030) 0:04:21.741 ******* 2022-01-11T13:56:05.187 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2022-01-11T13:56:05.189 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.213 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.032) 0:04:21.773 ******* 2022-01-11T13:56:05.221 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.245 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.031) 0:04:21.805 ******* 2022-01-11T13:56:05.253 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2022-01-11T13:56:05.255 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.279 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.033) 0:04:21.839 ******* 2022-01-11T13:56:05.287 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.311 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.031) 0:04:21.871 ******* 2022-01-11T13:56:05.318 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2022-01-11T13:56:05.319 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.344 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.032) 0:04:21.904 ******* 2022-01-11T13:56:05.351 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.374 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.030) 0:04:21.934 ******* 2022-01-11T13:56:05.380 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2022-01-11T13:56:05.382 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.406 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.031) 0:04:21.966 ******* 2022-01-11T13:56:05.413 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.436 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.030) 0:04:21.996 ******* 2022-01-11T13:56:05.443 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2022-01-11T13:56:05.444 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.475 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.039) 0:04:22.035 ******* 2022-01-11T13:56:05.483 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.506 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.030) 0:04:22.066 ******* 2022-01-11T13:56:05.512 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2022-01-11T13:56:05.514 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.537 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.030) 0:04:22.097 ******* 2022-01-11T13:56:05.543 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.566 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.029) 0:04:22.126 ******* 2022-01-11T13:56:05.573 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2022-01-11T13:56:05.575 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.600 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.033) 0:04:22.160 ******* 2022-01-11T13:56:05.608 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.631 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.031) 0:04:22.191 ******* 2022-01-11T13:56:05.638 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2022-01-11T13:56:05.641 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.663 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.031) 0:04:22.223 ******* 2022-01-11T13:56:05.672 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.696 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.033) 0:04:22.256 ******* 2022-01-11T13:56:05.702 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2022-01-11T13:56:05.704 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.730 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.030) 0:04:22.287 ******* 2022-01-11T13:56:05.736 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.759 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.031) 0:04:22.319 ******* 2022-01-11T13:56:05.765 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2022-01-11T13:56:05.767 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.792 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.032) 0:04:22.352 ******* 2022-01-11T13:56:05.799 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.824 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.031) 0:04:22.384 ******* 2022-01-11T13:56:05.830 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2022-01-11T13:56:05.832 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.863 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.039) 0:04:22.423 ******* 2022-01-11T13:56:05.870 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.895 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.031) 0:04:22.455 ******* 2022-01-11T13:56:05.900 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2022-01-11T13:56:05.902 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.926 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.031) 0:04:22.486 ******* 2022-01-11T13:56:05.933 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:05.957 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:05 +0000 (0:00:00.030) 0:04:22.517 ******* 2022-01-11T13:56:06.144 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2022-01-11T13:56:06.146 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2022-01-11T13:56:06.167 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:06 +0000 (0:00:00.210) 0:04:22.727 ******* 2022-01-11T13:56:06.186 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2022-01-11T13:56:06.212 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:06 +0000 (0:00:00.044) 0:04:22.772 ******* 2022-01-11T13:56:06.375 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2022-01-11T13:56:06.377 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2022-01-11T13:56:06.399 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:06 +0000 (0:00:00.187) 0:04:22.959 ******* 2022-01-11T13:56:06.440 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2022-01-11T13:56:06.464 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:06 +0000 (0:00:00.064) 0:04:23.024 ******* 2022-01-11T13:56:06.601 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2022-01-11T13:56:06.603 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2022-01-11T13:56:06.637 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:06 +0000 (0:00:00.173) 0:04:23.197 ******* 2022-01-11T13:56:06.672 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2022-01-11T13:56:06.673 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:06.695 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:06 +0000 (0:00:00.057) 0:04:23.255 ******* 2022-01-11T13:56:06.717 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:06 +0000 (0:00:00.021) 0:04:23.277 ******* 2022-01-11T13:56:06.725 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2022-01-11T13:56:06.726 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:06.754 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:06 +0000 (0:00:00.037) 0:04:23.314 ******* 2022-01-11T13:56:06.761 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2022-01-11T13:56:06.763 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:06.786 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:06 +0000 (0:00:00.031) 0:04:23.346 ******* 2022-01-11T13:56:06.793 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:06.982 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:06 +0000 (0:00:00.196) 0:04:23.542 ******* 2022-01-11T13:56:06.990 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2022-01-11T13:56:06.992 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.015 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.032) 0:04:23.575 ******* 2022-01-11T13:56:07.022 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2022-01-11T13:56:07.023 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.047 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.032) 0:04:23.607 ******* 2022-01-11T13:56:07.054 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2022-01-11T13:56:07.056 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.079 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.031) 0:04:23.639 ******* 2022-01-11T13:56:07.084 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2022-01-11T13:56:07.086 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.110 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.030) 0:04:23.670 ******* 2022-01-11T13:56:07.127 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2022-01-11T13:56:07.129 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.150 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.040) 0:04:23.710 ******* 2022-01-11T13:56:07.156 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2022-01-11T13:56:07.157 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.180 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.030) 0:04:23.740 ******* 2022-01-11T13:56:07.186 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2022-01-11T13:56:07.188 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.210 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.030) 0:04:23.770 ******* 2022-01-11T13:56:07.217 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2022-01-11T13:56:07.219 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.241 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.030) 0:04:23.801 ******* 2022-01-11T13:56:07.248 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2022-01-11T13:56:07.249 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.272 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.031) 0:04:23.832 ******* 2022-01-11T13:56:07.279 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2022-01-11T13:56:07.281 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.304 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.031) 0:04:23.864 ******* 2022-01-11T13:56:07.310 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2022-01-11T13:56:07.312 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.334 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.030) 0:04:23.894 ******* 2022-01-11T13:56:07.341 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2022-01-11T13:56:07.342 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.364 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.030) 0:04:23.924 ******* 2022-01-11T13:56:07.371 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2022-01-11T13:56:07.373 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.396 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.031) 0:04:23.956 ******* 2022-01-11T13:56:07.402 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2022-01-11T13:56:07.404 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.429 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.033) 0:04:23.989 ******* 2022-01-11T13:56:07.436 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2022-01-11T13:56:07.438 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.462 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.032) 0:04:24.022 ******* 2022-01-11T13:56:07.476 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2022-01-11T13:56:07.478 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.500 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.038) 0:04:24.061 ******* 2022-01-11T13:56:07.508 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2022-01-11T13:56:07.510 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.532 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.031) 0:04:24.092 ******* 2022-01-11T13:56:07.538 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2022-01-11T13:56:07.540 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.562 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.029) 0:04:24.122 ******* 2022-01-11T13:56:07.568 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2022-01-11T13:56:07.570 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.592 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.030) 0:04:24.152 ******* 2022-01-11T13:56:07.599 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2022-01-11T13:56:07.601 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.625 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.032) 0:04:24.185 ******* 2022-01-11T13:56:07.632 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2022-01-11T13:56:07.635 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.658 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.033) 0:04:24.218 ******* 2022-01-11T13:56:07.666 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.691 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.032) 0:04:24.251 ******* 2022-01-11T13:56:07.699 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2022-01-11T13:56:07.701 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.728 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.036) 0:04:24.288 ******* 2022-01-11T13:56:07.736 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2022-01-11T13:56:07.737 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.761 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.032) 0:04:24.321 ******* 2022-01-11T13:56:07.768 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2022-01-11T13:56:07.770 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.794 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.032) 0:04:24.354 ******* 2022-01-11T13:56:07.801 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2022-01-11T13:56:07.803 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.826 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.032) 0:04:24.386 ******* 2022-01-11T13:56:07.833 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2022-01-11T13:56:07.835 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.859 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.033) 0:04:24.419 ******* 2022-01-11T13:56:07.867 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2022-01-11T13:56:07.869 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.893 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.033) 0:04:24.453 ******* 2022-01-11T13:56:07.901 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2022-01-11T13:56:07.903 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.926 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.033) 0:04:24.486 ******* 2022-01-11T13:56:07.933 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2022-01-11T13:56:07.935 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.958 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.031) 0:04:24.518 ******* 2022-01-11T13:56:07.967 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2022-01-11T13:56:07.968 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:07.992 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:07 +0000 (0:00:00.034) 0:04:24.552 ******* 2022-01-11T13:56:07.999 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2022-01-11T13:56:08.000 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.023 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.030) 0:04:24.583 ******* 2022-01-11T13:56:08.029 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2022-01-11T13:56:08.031 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.053 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.030) 0:04:24.613 ******* 2022-01-11T13:56:08.060 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2022-01-11T13:56:08.061 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.084 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.030) 0:04:24.644 ******* 2022-01-11T13:56:08.090 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.114 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.030) 0:04:24.675 ******* 2022-01-11T13:56:08.121 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2022-01-11T13:56:08.123 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.145 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.030) 0:04:24.706 ******* 2022-01-11T13:56:08.152 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2022-01-11T13:56:08.154 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.177 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.031) 0:04:24.737 ******* 2022-01-11T13:56:08.184 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2022-01-11T13:56:08.186 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.209 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.032) 0:04:24.769 ******* 2022-01-11T13:56:08.216 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2022-01-11T13:56:08.217 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.241 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.031) 0:04:24.801 ******* 2022-01-11T13:56:08.248 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2022-01-11T13:56:08.250 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2022-01-11T13:56:08.272 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.031) 0:04:24.832 ******* 2022-01-11T13:56:08.278 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2022-01-11T13:56:08.280 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.305 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.032) 0:04:24.865 ******* 2022-01-11T13:56:08.325 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi161.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} ok: [smithi161.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2022-01-11T13:56:08.335 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.337 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.347 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2022-01-11T13:56:08.350 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi161.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.379 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.073) 0:04:24.938 ******* 2022-01-11T13:56:08.389 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.423 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.044) 0:04:24.983 ******* 2022-01-11T13:56:08.430 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.474 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2022-01-11T13:56:08.476 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.505 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.082) 0:04:25.065 ******* 2022-01-11T13:56:08.639 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2022-01-11T13:56:08.641 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003912", "end": "2022-01-11 13:56:08.615789", "rc": 0, "start": "2022-01-11 13:56:08.611877", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/bin/teuthology-lock"]} 2022-01-11T13:56:08.672 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.166) 0:04:25.232 ******* 2022-01-11T13:56:08.700 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2022-01-11T13:56:08.701 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.725 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.053) 0:04:25.285 ******* 2022-01-11T13:56:08.732 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2022-01-11T13:56:08.733 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003276", "end": "2022-01-11 13:56:08.696811", "rc": 0, "start": "2022-01-11 13:56:08.693535", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/bin/teuthology-lock"]} 2022-01-11T13:56:08.759 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.033) 0:04:25.319 ******* 2022-01-11T13:56:08.765 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2022-01-11T13:56:08.766 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.790 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.030) 0:04:25.350 ******* 2022-01-11T13:56:08.796 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2022-01-11T13:56:08.798 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.822 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.032) 0:04:25.383 ******* 2022-01-11T13:56:08.829 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2022-01-11T13:56:08.830 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.853 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.030) 0:04:25.413 ******* 2022-01-11T13:56:08.860 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2022-01-11T13:56:08.862 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.885 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.031) 0:04:25.445 ******* 2022-01-11T13:56:08.892 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2022-01-11T13:56:08.894 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.917 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.031) 0:04:25.477 ******* 2022-01-11T13:56:08.925 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2022-01-11T13:56:08.925 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:08.952 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:08 +0000 (0:00:00.034) 0:04:25.512 ******* 2022-01-11T13:56:08.979 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2022-01-11T13:56:08.981 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:09.009 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:09 +0000 (0:00:00.056) 0:04:25.568 ******* 2022-01-11T13:56:10.447 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2022-01-11T13:56:10.449 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:56:10.473 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:10 +0000 (0:00:01.465) 0:04:27.033 ******* 2022-01-11T13:56:10.528 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi161.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930256158720, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227113320, 'block_used': 12981788, 'inode_total': 61054976, 'inode_available': 60985806, 'inode_used': 69170, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227113320, "block_size": 4096, "block_total": 240095108, "block_used": 12981788, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985806, "inode_total": 61054976, "inode_used": 69170, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930256158720, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}} 2022-01-11T13:56:10.548 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:10 +0000 (0:00:00.074) 0:04:27.108 ******* 2022-01-11T13:56:10.743 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2022-01-11T13:56:10.745 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005329", "end": "2022-01-11 13:56:10.725463", "rc": 0, "start": "2022-01-11 13:56:10.720134", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:10.770 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:10 +0000 (0:00:00.221) 0:04:27.330 ******* 2022-01-11T13:56:10.821 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi161.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930256158720, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227113320, 'block_used': 12981788, 'inode_total': 61054976, 'inode_available': 60985806, 'inode_used': 69170, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227113320, "block_size": 4096, "block_total": 240095108, "block_used": 12981788, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985806, "inode_total": 61054976, "inode_used": 69170, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930256158720, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:10.844 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:10 +0000 (0:00:00.072) 0:04:27.403 ******* 2022-01-11T13:56:10.870 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2022-01-11T13:56:10.893 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:10 +0000 (0:00:00.050) 0:04:27.453 ******* 2022-01-11T13:56:10.912 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2022-01-11T13:56:10.913 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:56:10.937 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:10 +0000 (0:00:00.043) 0:04:27.496 ******* 2022-01-11T13:56:10.987 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2022-01-11T13:56:10.989 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930256158720, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227113320, 'block_used': 12981788, 'inode_total': 61054976, 'inode_available': 60985806, 'inode_used': 69170, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227113320, "block_size": 4096, "block_total": 240095108, "block_used": 12981788, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985806, "inode_total": 61054976, "inode_used": 69170, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930256158720, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}} 2022-01-11T13:56:11.012 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:11 +0000 (0:00:00.075) 0:04:27.572 ******* 2022-01-11T13:56:11.211 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2022-01-11T13:56:11.213 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005204", "end": "2022-01-11 13:56:11.193501", "rc": 0, "start": "2022-01-11 13:56:11.188297", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:11.235 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:11 +0000 (0:00:00.223) 0:04:27.795 ******* 2022-01-11T13:56:11.285 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2022-01-11T13:56:11.287 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930256158720, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227113320, 'block_used': 12981788, 'inode_total': 61054976, 'inode_available': 60985806, 'inode_used': 69170, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227113320, "block_size": 4096, "block_total": 240095108, "block_used": 12981788, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985806, "inode_total": 61054976, "inode_used": 69170, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930256158720, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:11.312 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:11 +0000 (0:00:00.077) 0:04:27.872 ******* 2022-01-11T13:56:11.343 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2022-01-11T13:56:11.379 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:11 +0000 (0:00:00.066) 0:04:27.939 ******* 2022-01-11T13:56:12.520 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2022-01-11T13:56:12.522 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8XgU0OVhMTgV21Ax15KAyvGeGAJqzpndbL'], 'uuids': ['fd034af0-bd62-4fd4-8af9-8e93e5adb3cc'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.409472", "end": "2022-01-11 13:56:12.491964", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8XgU0OVhMTgV21Ax15KAyvGeGAJqzpndbL"], "labels": [], "masters": [], "uuids": ["fd034af0-bd62-4fd4-8af9-8e93e5adb3cc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:11.082492", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2022-01-11T13:56:12.991 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4m78ESXDKqzuLFIlb2oPVHAHcyexF9dAK'], 'uuids': ['af698db4-1f9f-4f53-85cb-6d2ed35607c9'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.381191", "end": "2022-01-11 13:56:12.956652", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4m78ESXDKqzuLFIlb2oPVHAHcyexF9dAK"], "labels": [], "masters": [], "uuids": ["af698db4-1f9f-4f53-85cb-6d2ed35607c9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:11.575461", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2022-01-11T13:56:13.789 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-HoaJWR-qVCQ-xyti-3Muh-cSum-8uOc-nPzEVt', 'nvme-INTEL_SSDPEDMD400G4_CVFT6233001X400BGN', 'nvme-nvme.8086-43564654363233333030315834303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.074902", "end": "2022-01-11 13:56:13.758195", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-HoaJWR-qVCQ-xyti-3Muh-cSum-8uOc-nPzEVt", "nvme-INTEL_SSDPEDMD400G4_CVFT6233001X400BGN", "nvme-nvme.8086-43564654363233333030315834303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:12.683293", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2022-01-11T13:56:13.791 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2022-01-11T13:56:14.228 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-ohtWBO-FJGC-Tdea-KcuR-oRDz-VKhw-mxjm50', 'nvme-INTEL_SSDPEDMD400G4_CVFT6233005M400BGN', 'nvme-nvme.8086-43564654363233333030354d34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.050556", "end": "2022-01-11 13:56:14.198706", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ohtWBO-FJGC-Tdea-KcuR-oRDz-VKhw-mxjm50", "nvme-INTEL_SSDPEDMD400G4_CVFT6233005M400BGN", "nvme-nvme.8086-43564654363233333030354d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:13.148150", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2022-01-11T13:56:14.230 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2022-01-11T13:56:15.037 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8XUEyLTCTiZJko9HehU03kJORKX5R5Oekk'], 'uuids': ['6202bfca-e892-40ef-af02-09236ded6e12'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.062882", "end": "2022-01-11 13:56:15.008229", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8XUEyLTCTiZJko9HehU03kJORKX5R5Oekk"], "labels": [], "masters": [], "uuids": ["6202bfca-e892-40ef-af02-09236ded6e12"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:13.945347", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2022-01-11T13:56:15.467 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4WfSP6ZB5kBZJrZuwBWCcHdV7C46aQjDT'], 'uuids': ['8a296386-70d1-4ee1-a1ab-a0f12bb6dcde'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.052203", "end": "2022-01-11 13:56:15.440227", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4WfSP6ZB5kBZJrZuwBWCcHdV7C46aQjDT"], "labels": [], "masters": [], "uuids": ["8a296386-70d1-4ee1-a1ab-a0f12bb6dcde"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:14.388024", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2022-01-11T13:56:16.284 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8X0ro7OB0Fo78sqDnfAq552mnRyU3u27WK'], 'uuids': ['cab3e094-a04c-45ec-a727-06af6eb32ad9'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.062525", "end": "2022-01-11 13:56:16.259643", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8X0ro7OB0Fo78sqDnfAq552mnRyU3u27WK"], "labels": [], "masters": [], "uuids": ["cab3e094-a04c-45ec-a727-06af6eb32ad9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:15.197118", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2022-01-11T13:56:16.713 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4x2Pf5nbAkDOhDmlnfdS3CJ173VCk0Okx'], 'uuids': ['eab52b1d-d9bf-4fcc-acd9-6f672504a8a2'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.051863", "end": "2022-01-11 13:56:16.683807", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4x2Pf5nbAkDOhDmlnfdS3CJ173VCk0Okx"], "labels": [], "masters": [], "uuids": ["eab52b1d-d9bf-4fcc-acd9-6f672504a8a2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:15.631944", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2022-01-11T13:56:17.574 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8XinCIiyewBP387cUt1fqkcU14FD7xzU3c'], 'uuids': ['32f690db-1121-4410-9483-cf2c49f036ec'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.088448", "end": "2022-01-11 13:56:17.550048", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8XinCIiyewBP387cUt1fqkcU14FD7xzU3c"], "labels": [], "masters": [], "uuids": ["32f690db-1121-4410-9483-cf2c49f036ec"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:16.461600", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2022-01-11T13:56:17.586 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XV0E', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XV0E', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XV0E', 'scsi-35000c50092938c4f', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XV0E', 'wwn-0x5000c50092938c4f'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XV0E', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50092938c4f', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XV0E-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XV0E-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XV0E-part1', 'scsi-35000c50092938c4f-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XV0E-part1', 'wwn-0x5000c50092938c4f-part1'], 'uuids': ['fcc12580-8f3b-4cec-aef7-e7427fa239b7'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XV0E", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XV0E", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XV0E", "scsi-35000c50092938c4f", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XV0E", "wwn-0x5000c50092938c4f"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XV0E-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XV0E-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2022-01-11T13:56:17.588 INFO:teuthology.task.ansible.out:Z1W5XV0E-part1", "scsi-35000c50092938c4f-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XV0E-part1", "wwn-0x5000c50092938c4f-part1"], "labels": [], "masters": [], "uuids": ["fcc12580-8f3b-4cec-aef7-e7427fa239b7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XV0E", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092938c4f"}}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:17.958 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4C5Ed1ZidFd75KHiAoGTz3spAwCqdrcff'], 'uuids': ['2fe8e33f-cc77-446c-9d29-42fc2c70ff56'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.065521", "end": "2022-01-11 13:56:17.932557", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4C5Ed1ZidFd75KHiAoGTz3spAwCqdrcff"], "labels": [], "masters": [], "uuids": ["2fe8e33f-cc77-446c-9d29-42fc2c70ff56"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:16.867036", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2022-01-11T13:56:17.970 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNKA', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNKA', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNKA', 'scsi-35000c5009294065e', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XNKA', 'wwn-0x5000c5009294065e'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XNKA', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009294065e', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNKA-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNKA-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNKA-part1', 'scsi-35000c5009294065e-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XNKA-part1', 'wwn-0x5000c5009294065e-part1'], 'uuids': ['fcc12580-8f3b-4cec-aef7-e7427fa239b7'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNKA", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNKA", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNKA", "scsi-35000c5009294065e", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XNKA", "wwn-0x5000c5009294065e"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNKA-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNKA-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2022-01-11T13:56:17.972 INFO:teuthology.task.ansible.out:Z1W5XNKA-part1", "scsi-35000c5009294065e-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XNKA-part1", "wwn-0x5000c5009294065e-part1"], "labels": [], "masters": [], "uuids": ["fcc12580-8f3b-4cec-aef7-e7427fa239b7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNKA", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294065e"}}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:18.855 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8X7LRMS0WcU5ec6RdgqNL23raMeW41ujsI'], 'uuids': ['2346e1db-80e7-491d-8695-a16a64c857c1'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.082905", "end": "2022-01-11 13:56:18.830186", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8X7LRMS0WcU5ec6RdgqNL23raMeW41ujsI"], "labels": [], "masters": [], "uuids": ["2346e1db-80e7-491d-8695-a16a64c857c1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:17.747281", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2022-01-11T13:56:18.897 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:18 +0000 (0:00:07.517) 0:04:35.457 ******* 2022-01-11T13:56:19.218 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4SeZRr7LafpAmSyiG2Bsr1ZntuV9xKCad'], 'uuids': ['d4401183-b7b3-495a-93dd-28e5d94096ee'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.069579", "end": "2022-01-11 13:56:19.192266", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4SeZRr7LafpAmSyiG2Bsr1ZntuV9xKCad"], "labels": [], "masters": [], "uuids": ["d4401183-b7b3-495a-93dd-28e5d94096ee"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:18.122687", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2022-01-11T13:56:19.258 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:19 +0000 (0:00:00.360) 0:04:35.818 ******* 2022-01-11T13:56:19.275 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2022-01-11T13:56:19.276 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8XgU0OVhMTgV21Ax15KAyvGeGAJqzpndbL'], 'uuids': ['fd034af0-bd62-4fd4-8af9-8e93e5adb3cc'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.144459", "end": "2022-01-11 13:56:19.243844", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8XgU0OVhMTgV21Ax15KAyvGeGAJqzpndbL"], "labels": [], "masters": [], "uuids": ["fd034af0-bd62-4fd4-8af9-8e93e5adb3cc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:19.099385", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141707 s, 814 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141707 s, 814 MB/s"], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:19.487 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-HoaJWR-qVCQ-xyti-3Muh-cSum-8uOc-nPzEVt', 'nvme-INTEL_SSDPEDMD400G4_CVFT6233001X400BGN', 'nvme-nvme.8086-43564654363233333030315834303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039918", "end": "2022-01-11 13:56:19.460903", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-HoaJWR-qVCQ-xyti-3Muh-cSum-8uOc-nPzEVt", "nvme-INTEL_SSDPEDMD400G4_CVFT6233001X400BGN", "nvme-nvme.8086-43564654363233333030315834303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:19.420985", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0371312 s, 3.1 GB/s", "std 2022-01-11T13:56:19.489 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0371312 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:19.625 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4m78ESXDKqzuLFIlb2oPVHAHcyexF9dAK'], 'uuids': ['af698db4-1f9f-4f53-85cb-6d2ed35607c9'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.142747", "end": "2022-01-11 13:56:19.600871", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4m78ESXDKqzuLFIlb2oPVHAHcyexF9dAK"], "labels": [], "masters": [], "uuids": ["af698db4-1f9f-4f53-85cb-6d2ed35607c9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:19.458124", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139979 s, 824 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139979 s, 824 MB/s"], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:19.823 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8XUEyLTCTiZJko9HehU03kJORKX5R5Oekk'], 'uuids': ['6202bfca-e892-40ef-af02-09236ded6e12'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.145507", "end": "2022-01-11 13:56:19.788004", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8XUEyLTCTiZJko9HehU03kJORKX5R5Oekk"], "labels": [], "masters": [], "uuids": ["6202bfca-e892-40ef-af02-09236ded6e12"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:19.642497", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142905 s, 807 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142905 s, 807 MB/s"], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:19.835 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-ohtWBO-FJGC-Tdea-KcuR-oRDz-VKhw-mxjm50', 'nvme-INTEL_SSDPEDMD400G4_CVFT6233005M400BGN', 'nvme-nvme.8086-43564654363233333030354d34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039096", "end": "2022-01-11 13:56:19.816458", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ohtWBO-FJGC-Tdea-KcuR-oRDz-VKhw-mxjm50", "nvme-INTEL_SSDPEDMD400G4_CVFT6233005M400BGN", "nvme-nvme.8086-43564654363233333030354d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:19.777362", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0363515 s, 3.2 GB/s", "std 2022-01-11T13:56:19.837 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0363515 s, 3.2 GB/s"], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:20.133 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8X0ro7OB0Fo78sqDnfAq552mnRyU3u27WK'], 'uuids': ['cab3e094-a04c-45ec-a727-06af6eb32ad9'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.143342", "end": "2022-01-11 13:56:20.102131", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8X0ro7OB0Fo78sqDnfAq552mnRyU3u27WK"], "labels": [], "masters": [], "uuids": ["cab3e094-a04c-45ec-a727-06af6eb32ad9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:19.958789", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14079 s, 819 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14079 s, 819 MB/s"], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:20.549 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4WfSP6ZB5kBZJrZuwBWCcHdV7C46aQjDT'], 'uuids': ['8a296386-70d1-4ee1-a1ab-a0f12bb6dcde'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.143811", "end": "2022-01-11 13:56:20.144354", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4WfSP6ZB5kBZJrZuwBWCcHdV7C46aQjDT"], "labels": [], "masters": [], "uuids": ["8a296386-70d1-4ee1-a1ab-a0f12bb6dcde"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:20.000543", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141091 s, 818 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141091 s, 818 MB/s"], "stdout": "", "stdout_lines": []} changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8XinCIiyewBP387cUt1fqkcU14FD7xzU3c'], 'uuids': ['32f690db-1121-4410-9483-cf2c49f036ec'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '51 2022-01-11T13:56:20.550 INFO:teuthology.task.ansible.out:2', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.153548", "end": "2022-01-11 13:56:20.439128", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8XinCIiyewBP387cUt1fqkcU14FD7xzU3c"], "labels": [], "masters": [], "uuids": ["32f690db-1121-4410-9483-cf2c49f036ec"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:20.285580", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150952 s, 764 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150952 s, 764 MB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi161.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XV0E', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XV0E', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XV0E', 'scsi-35000c50092938c4f', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XV0E', 'wwn-0x5000c50092938c4f'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XV0E', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50092938c4f', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XV0E-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XV0E-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XV0E-part1', 'scsi-35000c50092938c4f-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XV0E-part1', 'wwn-0x5000c50092938c4f- 2022-01-11T13:56:20.552 INFO:teuthology.task.ansible.out:part1'], 'uuids': ['fcc12580-8f3b-4cec-aef7-e7427fa239b7'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XV0E", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XV0E", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XV0E", "scsi-35000c50092938c4f", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XV0E", "wwn-0x5000c50092938c4f"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XV0E-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XV0E-part1", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XV0E-part1", "scsi-35000c50092938c4f-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XV0E-part1", "wwn-0x5000c50092938c4f-part1"], "labels": [], "masters": [], "uuids": ["fcc12580-8f3b-4cec-aef7-e7427fa239b7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XV0E", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092938c4f"}}, "skip_reason": "Conditional result was False"} changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM 2022-01-11T13:56:20.554 INFO:teuthology.task.ansible.out:-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4x2Pf5nbAkDOhDmlnfdS3CJ173VCk0Okx'], 'uuids': ['eab52b1d-d9bf-4fcc-acd9-6f672504a8a2'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.145140", "end": "2022-01-11 13:56:20.472658", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4x2Pf5nbAkDOhDmlnfdS3CJ173VCk0Okx"], "labels": [], "masters": [], "uuids": ["eab52b1d-d9bf-4fcc-acd9-6f672504a8a2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:20.327518", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14253 s, 809 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14253 s, 809 MB/s"], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:20.786 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8X7LRMS0WcU5ec6RdgqNL23raMeW41ujsI'], 'uuids': ['2346e1db-80e7-491d-8695-a16a64c857c1'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.141882", "end": "2022-01-11 13:56:20.765257", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-eWfuv4z1aRtsmXAJXXf3xbsRE8yQPd8X7LRMS0WcU5ec6RdgqNL23raMeW41ujsI"], "labels": [], "masters": [], "uuids": ["2346e1db-80e7-491d-8695-a16a64c857c1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:20.623375", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139228 s, 828 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139228 s, 828 MB/s"], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:20.816 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:20 +0000 (0:00:01.558) 0:04:37.376 ******* 2022-01-11T13:56:20.825 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4C5Ed1ZidFd75KHiAoGTz3spAwCqdrcff'], 'uuids': ['2fe8e33f-cc77-446c-9d29-42fc2c70ff56'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.143131", "end": "2022-01-11 13:56:20.793011", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4C5Ed1ZidFd75KHiAoGTz3spAwCqdrcff"], "labels": [], "masters": [], "uuids": ["2fe8e33f-cc77-446c-9d29-42fc2c70ff56"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:20.649880", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140536 s, 821 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140536 s, 821 MB/s"], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:20.827 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNKA', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNKA', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNKA', 'scsi-35000c5009294065e', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XNKA', 'wwn-0x5000c5009294065e'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XNKA', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009294065e', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNKA-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNKA-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNKA-part1', 'scsi-35000c5009294065e-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XNKA-part1', 'wwn-0x5000c5009294065e-part1'], 'uuids': ['fcc12580-8f3b-4cec-aef7-e7427fa239b7'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNKA", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNKA", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNKA", "scsi-35000c5009294065e", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XNKA", "wwn-0x5000c5009294065e"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNKA-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNKA-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2022-01-11T13:56:20.828 INFO:teuthology.task.ansible.out:Z1W5XNKA-part1", "scsi-35000c5009294065e-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XNKA-part1", "wwn-0x5000c5009294065e-part1"], "labels": [], "masters": [], "uuids": ["fcc12580-8f3b-4cec-aef7-e7427fa239b7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNKA", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294065e"}}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:21.070 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi161.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.064821", "end": "2022-01-11 13:56:21.041746", "rc": 0, "start": "2022-01-11 13:56:20.976925", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:21.093 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.276) 0:04:37.653 ******* 2022-01-11T13:56:21.142 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2022-01-11T13:56:21.144 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4SeZRr7LafpAmSyiG2Bsr1ZntuV9xKCad'], 'uuids': ['d4401183-b7b3-495a-93dd-28e5d94096ee'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.141563", "end": "2022-01-11 13:56:21.113298", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-xv2ZQfdKLsfUzyVJK66Cwhf6jyCwdyZ4SeZRr7LafpAmSyiG2Bsr1ZntuV9xKCad"], "labels": [], "masters": [], "uuids": ["d4401183-b7b3-495a-93dd-28e5d94096ee"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2022-01-11 13:56:20.971735", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139057 s, 829 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139057 s, 829 MB/s"], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:21.177 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.084) 0:04:37.737 ******* 2022-01-11T13:56:21.331 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2022-01-11T13:56:21.332 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.051600", "end": "2022-01-11 13:56:21.307666", "rc": 0, "start": "2022-01-11 13:56:21.256066", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:21.355 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.178) 0:04:37.915 ******* 2022-01-11T13:56:21.384 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2022-01-11T13:56:21.411 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.055) 0:04:37.971 ******* 2022-01-11T13:56:21.429 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2022-01-11T13:56:21.431 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.060089", "end": "2022-01-11 13:56:21.400094", "rc": 0, "start": "2022-01-11 13:56:21.340005", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:21.462 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.041) 0:04:38.013 ******* TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:21.490 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.037) 0:04:38.050 ******* 2022-01-11T13:56:21.531 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:21.554 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.064) 0:04:38.114 ******* 2022-01-11T13:56:21.588 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:21.609 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.055) 0:04:38.169 ******* 2022-01-11T13:56:21.638 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2022-01-11T13:56:21.640 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:21.663 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.053) 0:04:38.223 ******* 2022-01-11T13:56:21.702 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2022-01-11T13:56:21.704 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.060600", "end": "2022-01-11 13:56:21.679975", "rc": 0, "start": "2022-01-11 13:56:21.619375", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:21.733 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.070) 0:04:38.293 ******* 2022-01-11T13:56:21.762 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2022-01-11T13:56:21.789 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.055) 0:04:38.349 ******* 2022-01-11T13:56:21.828 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2022-01-11T13:56:21.829 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:21.853 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.063) 0:04:38.413 ******* 2022-01-11T13:56:21.859 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2022-01-11T13:56:21.861 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2022-01-11T13:56:21.884 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.031) 0:04:38.444 ******* 2022-01-11T13:56:21.891 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2022-01-11T13:56:21.893 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:21.915 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.030) 0:04:38.475 ******* 2022-01-11T13:56:21.917 INFO:teuthology.task.ansible.out: 2022-01-11T13:56:21.933 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi161.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930256158720, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227113320, 'block_used': 12981788, 'inode_total': 61054976, 'inode_available': 60985806, 'inode_used': 69170, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227113320, "block_size": 4096, "block_total": 240095108, "block_used": 12981788, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985806, "inode_total": 61054976, "inode_used": 69170, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930256158720, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:21.958 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.042) 0:04:38.518 ******* 2022-01-11T13:56:21.967 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2022-01-11T13:56:21.969 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:21.995 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:21 +0000 (0:00:00.037) 0:04:38.556 ******* 2022-01-11T13:56:22.002 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2022-01-11T13:56:22.004 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:22.028 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.032) 0:04:38.588 ******* 2022-01-11T13:56:22.046 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2022-01-11T13:56:22.048 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:22.069 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.040) 0:04:38.629 ******* 2022-01-11T13:56:22.076 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2022-01-11T13:56:22.078 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:22.101 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.031) 0:04:38.661 ******* 2022-01-11T13:56:22.129 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2022-01-11T13:56:22.130 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:22.155 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.053) 0:04:38.715 ******* 2022-01-11T13:56:22.183 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2022-01-11T13:56:22.185 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:22.207 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.052) 0:04:38.767 ******* 2022-01-11T13:56:22.235 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2022-01-11T13:56:22.237 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:22.259 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.051) 0:04:38.819 ******* 2022-01-11T13:56:22.266 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2022-01-11T13:56:22.267 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2022-01-11T13:56:22.292 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.032) 0:04:38.852 ******* 2022-01-11T13:56:22.299 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2022-01-11T13:56:22.301 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:22.324 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.032) 0:04:38.884 ******* 2022-01-11T13:56:22.343 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2022-01-11T13:56:22.345 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930256158720, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227113320, 'block_used': 12981788, 'inode_total': 61054976, 'inode_available': 60985806, 'inode_used': 69170, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227113320, "block_size": 4096, "block_total": 240095108, "block_used": 12981788, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985806, "inode_total": 61054976, "inode_used": 69170, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930256158720, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:22.369 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.044) 0:04:38.929 ******* 2022-01-11T13:56:22.409 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2022-01-11T13:56:22.410 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:22.433 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.064) 0:04:38.993 ******* 2022-01-11T13:56:22.463 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2022-01-11T13:56:22.465 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:22.486 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.053) 0:04:39.046 ******* 2022-01-11T13:56:22.514 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:22.541 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.054) 0:04:39.101 ******* 2022-01-11T13:56:22.582 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:22.606 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.064) 0:04:39.166 ******* 2022-01-11T13:56:22.634 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2022-01-11T13:56:22.636 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:22.657 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.051) 0:04:39.218 ******* 2022-01-11T13:56:22.695 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:22.719 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.061) 0:04:39.279 ******* 2022-01-11T13:56:22.861 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2022-01-11T13:56:22.863 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2022-01-11T13:56:22.886 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:22 +0000 (0:00:00.166) 0:04:39.445 ******* 2022-01-11T13:56:23.165 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2022-01-11T13:56:23.188 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:23 +0000 (0:00:00.302) 0:04:39.748 ******* 2022-01-11T13:56:23.340 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2022-01-11T13:56:23.342 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2022-01-11T13:56:23.605 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2022-01-11T13:56:23.678 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2022-01-11T13:56:23.940 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2022-01-11T13:56:24.014 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2022-01-11T13:56:24.268 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2022-01-11T13:56:24.362 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2022-01-11T13:56:24.612 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2022-01-11T13:56:24.706 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2022-01-11T13:56:24.731 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:24 +0000 (0:00:01.542) 0:04:41.291 ******* 2022-01-11T13:56:24.925 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2022-01-11T13:56:24.951 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:24 +0000 (0:00:00.220) 0:04:41.511 ******* 2022-01-11T13:56:25.000 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2022-01-11T13:56:25.002 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2022-01-11T13:56:25.026 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:25 +0000 (0:00:00.074) 0:04:41.586 ******* 2022-01-11T13:56:25.163 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2022-01-11T13:56:25.165 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2022-01-11T13:56:25.229 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2022-01-11T13:56:25.258 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:25 +0000 (0:00:00.232) 0:04:41.818 ******* 2022-01-11T13:56:25.343 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2022-01-11T13:56:25.478 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2022-01-11T13:56:25.538 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2022-01-11T13:56:25.660 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2022-01-11T13:56:25.725 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2022-01-11T13:56:25.728 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:25.773 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:25 +0000 (0:00:00.514) 0:04:42.333 ******* 2022-01-11T13:56:25.804 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2022-01-11T13:56:25.806 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:25.826 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:25 +0000 (0:00:00.053) 0:04:42.386 ******* 2022-01-11T13:56:25.843 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2022-01-11T13:56:25.846 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2022-01-11T13:56:25.855 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:25.878 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:25 +0000 (0:00:00.051) 0:04:42.438 ******* 2022-01-11T13:56:26.020 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2022-01-11T13:56:26.032 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:26.059 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:26 +0000 (0:00:00.181) 0:04:42.619 ******* 2022-01-11T13:56:26.086 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2022-01-11T13:56:26.088 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:26.109 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:26 +0000 (0:00:00.049) 0:04:42.669 ******* 2022-01-11T13:56:26.116 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2022-01-11T13:56:26.117 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2022-01-11T13:56:26.141 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:26 +0000 (0:00:00.032) 0:04:42.701 ******* 2022-01-11T13:56:26.148 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2022-01-11T13:56:26.150 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:26.173 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:26 +0000 (0:00:00.031) 0:04:42.733 ******* 2022-01-11T13:56:26.181 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2022-01-11T13:56:26.184 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2022-01-11T13:56:26.208 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:26 +0000 (0:00:00.035) 0:04:42.768 ******* 2022-01-11T13:56:26.376 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2022-01-11T13:56:26.378 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2022-01-11T13:56:26.400 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:26 +0000 (0:00:00.191) 0:04:42.960 ******* 2022-01-11T13:56:26.438 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2022-01-11T13:56:26.440 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2022-01-11T13:56:26.466 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:26 +0000 (0:00:00.066) 0:04:43.026 ******* 2022-01-11T13:56:26.630 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi161.front.sepia.ceph.com] => {"changed": true} 2022-01-11T13:56:26.653 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:26 +0000 (0:00:00.186) 0:04:43.213 ******* 2022-01-11T13:56:26.755 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true} 2022-01-11T13:56:26.778 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:26 +0000 (0:00:00.125) 0:04:43.338 ******* 2022-01-11T13:56:27.065 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2022-01-11T13:56:27.067 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2022-01-11T13:56:27.090 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.311) 0:04:43.650 ******* 2022-01-11T13:56:27.098 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2022-01-11T13:56:27.122 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.032) 0:04:43.682 ******* 2022-01-11T13:56:27.129 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2022-01-11T13:56:27.130 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.153 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.031) 0:04:43.714 ******* 2022-01-11T13:56:27.161 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.186 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.032) 0:04:43.746 ******* 2022-01-11T13:56:27.192 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2022-01-11T13:56:27.194 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.218 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.032) 0:04:43.778 ******* 2022-01-11T13:56:27.227 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.252 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.034) 0:04:43.812 ******* 2022-01-11T13:56:27.260 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2022-01-11T13:56:27.262 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.285 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.032) 0:04:43.845 ******* 2022-01-11T13:56:27.294 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.317 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.032) 0:04:43.878 ******* 2022-01-11T13:56:27.324 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2022-01-11T13:56:27.325 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.347 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.029) 0:04:43.907 ******* 2022-01-11T13:56:27.354 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.377 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.030) 0:04:43.937 ******* 2022-01-11T13:56:27.384 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2022-01-11T13:56:27.386 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.408 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.030) 0:04:43.968 ******* 2022-01-11T13:56:27.415 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.439 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.031) 0:04:43.999 ******* 2022-01-11T13:56:27.445 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2022-01-11T13:56:27.447 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.469 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.029) 0:04:44.029 ******* 2022-01-11T13:56:27.475 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.499 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.030) 0:04:44.059 ******* 2022-01-11T13:56:27.505 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2022-01-11T13:56:27.507 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.529 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.029) 0:04:44.089 ******* 2022-01-11T13:56:27.535 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.559 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.029) 0:04:44.119 ******* 2022-01-11T13:56:27.565 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2022-01-11T13:56:27.566 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.589 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.029) 0:04:44.149 ******* 2022-01-11T13:56:27.596 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.620 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.030) 0:04:44.180 ******* 2022-01-11T13:56:27.625 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2022-01-11T13:56:27.627 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.649 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.029) 0:04:44.209 ******* 2022-01-11T13:56:27.655 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.684 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.034) 0:04:44.244 ******* 2022-01-11T13:56:27.691 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2022-01-11T13:56:27.692 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.718 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.031) 0:04:44.275 ******* 2022-01-11T13:56:27.722 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.746 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.031) 0:04:44.306 ******* 2022-01-11T13:56:27.754 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2022-01-11T13:56:27.756 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.778 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.031) 0:04:44.338 ******* 2022-01-11T13:56:27.784 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.808 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.030) 0:04:44.368 ******* 2022-01-11T13:56:27.815 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2022-01-11T13:56:27.817 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.839 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.030) 0:04:44.399 ******* 2022-01-11T13:56:27.846 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.870 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.030) 0:04:44.430 ******* 2022-01-11T13:56:27.877 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2022-01-11T13:56:27.878 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.901 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.031) 0:04:44.461 ******* 2022-01-11T13:56:27.908 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.932 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.031) 0:04:44.492 ******* 2022-01-11T13:56:27.939 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2022-01-11T13:56:27.941 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.964 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.032) 0:04:44.524 ******* 2022-01-11T13:56:27.972 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:27.996 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:27 +0000 (0:00:00.031) 0:04:44.556 ******* 2022-01-11T13:56:28.003 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2022-01-11T13:56:28.004 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:28.026 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:28 +0000 (0:00:00.030) 0:04:44.586 ******* 2022-01-11T13:56:28.033 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:28.057 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:28 +0000 (0:00:00.031) 0:04:44.617 ******* 2022-01-11T13:56:28.512 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2022-01-11T13:56:28.514 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2022-01-11T13:56:28.536 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:28 +0000 (0:00:00.478) 0:04:45.096 ******* 2022-01-11T13:56:28.544 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2022-01-11T13:56:28.569 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:28 +0000 (0:00:00.033) 0:04:45.129 ******* 2022-01-11T13:56:28.846 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2022-01-11T13:56:28.847 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "900", "ExecMainStartTimestamp": "Tue 2022-01-11 13:50:41 UTC", "ExecMainStartTimestampMonotonic": "12473900", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSI 2022-01-11T13:56:28.849 INFO:teuthology.task.ansible.out:ZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "900", "MemoryAccounting": "yes", "MemoryCurrent": "1835008", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "forking", "UID": "[not set]", "UMask": "0022", "WatchdogTimestamp": "Tue 2022-01-11 1 2022-01-11T13:56:28.850 INFO:teuthology.task.ansible.out:3:50:41 UTC", "WatchdogTimestampMonotonic": "12473911", "WatchdogUSec": "0"}} 2022-01-11T13:56:28.871 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:28 +0000 (0:00:00.301) 0:04:45.431 ******* 2022-01-11T13:56:28.880 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "814", "ExecMainStartTimestamp": "Tue 2022-01-11 13:50:38 UTC", "ExecMainStartTimestampMonotonic": "11961955", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSI 2022-01-11T13:56:28.882 INFO:teuthology.task.ansible.out:ZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "814", "MemoryAccounting": "yes", "MemoryCurrent": "2076672", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "forking", "UID": "[not set]", "UMask": "0022", "WatchdogTimestamp": "Tue 2022-01-11 1 2022-01-11T13:56:28.884 INFO:teuthology.task.ansible.out:3:50:38 UTC", "WatchdogTimestampMonotonic": "11961966", "WatchdogUSec": "0"}} 2022-01-11T13:56:28.904 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:28 +0000 (0:00:00.033) 0:04:45.464 ******* 2022-01-11T13:56:28.911 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2022-01-11T13:56:28.913 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2022-01-11T13:56:28.935 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:28 +0000 (0:00:00.031) 0:04:45.495 ******* 2022-01-11T13:56:28.943 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2022-01-11T13:56:28.966 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:28 +0000 (0:00:00.030) 0:04:45.526 ******* 2022-01-11T13:56:28.973 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2022-01-11T13:56:28.975 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2022-01-11T13:56:29.001 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:28 +0000 (0:00:00.034) 0:04:45.561 ******* 2022-01-11T13:56:29.018 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2022-01-11T13:56:29.041 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:29 +0000 (0:00:00.039) 0:04:45.601 ******* 2022-01-11T13:56:29.047 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2022-01-11T13:56:29.049 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:29.072 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:29 +0000 (0:00:00.030) 0:04:45.632 ******* 2022-01-11T13:56:29.079 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:29.106 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:29 +0000 (0:00:00.034) 0:04:45.666 ******* 2022-01-11T13:56:29.112 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2022-01-11T13:56:29.114 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:29.137 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:29 +0000 (0:00:00.031) 0:04:45.697 ******* 2022-01-11T13:56:29.145 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:29.169 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:29 +0000 (0:00:00.031) 0:04:45.729 ******* 2022-01-11T13:56:29.177 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2022-01-11T13:56:29.178 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:29.202 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:29 +0000 (0:00:00.033) 0:04:45.762 ******* 2022-01-11T13:56:29.210 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:29.234 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:29 +0000 (0:00:00.031) 0:04:45.794 ******* 2022-01-11T13:56:29.406 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2022-01-11T13:56:29.408 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2022-01-11T13:56:29.431 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:29 +0000 (0:00:00.197) 0:04:45.991 ******* 2022-01-11T13:56:29.442 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2022-01-11T13:56:29.477 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:29 +0000 (0:00:00.045) 0:04:46.037 ******* 2022-01-11T13:56:29.912 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2022-01-11T13:56:29.914 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2022-01-11T13:56:29.948 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:29 +0000 (0:00:00.470) 0:04:46.508 ******* 2022-01-11T13:56:29.960 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2022-01-11T13:56:29.991 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:29 +0000 (0:00:00.042) 0:04:46.551 ******* 2022-01-11T13:56:31.115 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2022-01-11T13:56:31.117 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)"], "version": null, "virtualenv": null} 2022-01-11T13:56:31.150 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:31 +0000 (0:00:01.159) 0:04:47.711 ******* 2022-01-11T13:56:31.172 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)"], "version": null, "virtualenv": null} 2022-01-11T13:56:31.195 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:31 +0000 (0:00:00.044) 0:04:47.755 ******* 2022-01-11T13:56:31.202 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2022-01-11T13:56:31.205 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:31.233 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:31 +0000 (0:00:00.037) 0:04:47.793 ******* 2022-01-11T13:56:31.244 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:31.277 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:31 +0000 (0:00:00.043) 0:04:47.837 ******* 2022-01-11T13:56:31.287 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2022-01-11T13:56:31.289 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2022-01-11T13:56:31.314 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:31 +0000 (0:00:00.037) 0:04:47.874 ******* 2022-01-11T13:56:31.323 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2022-01-11T13:56:31.349 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:31 +0000 (0:00:00.034) 0:04:47.909 ******* 2022-01-11T13:56:31.506 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2022-01-11T13:56:31.508 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2022-01-11T13:56:31.538 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:31 +0000 (0:00:00.189) 0:04:48.098 ******* 2022-01-11T13:56:31.550 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2022-01-11T13:56:31.576 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:31 +0000 (0:00:00.038) 0:04:48.136 ******* 2022-01-11T13:56:31.584 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2022-01-11T13:56:31.586 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:31.625 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:31 +0000 (0:00:00.048) 0:04:48.185 ******* 2022-01-11T13:56:31.637 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:31.674 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:31 +0000 (0:00:00.049) 0:04:48.234 ******* 2022-01-11T13:56:31.683 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2022-01-11T13:56:31.685 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2022-01-11T13:56:31.717 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:31 +0000 (0:00:00.039) 0:04:48.274 ******* 2022-01-11T13:56:31.721 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2022-01-11T13:56:31.759 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:31 +0000 (0:00:00.045) 0:04:48.319 ******* 2022-01-11T13:56:31.773 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2022-01-11T13:56:31.775 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:31.800 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:31 +0000 (0:00:00.040) 0:04:48.360 ******* 2022-01-11T13:56:31.808 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:31.835 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:31 +0000 (0:00:00.035) 0:04:48.395 ******* 2022-01-11T13:56:31.992 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2022-01-11T13:56:31.994 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2022-01-11T13:56:32.016 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.181) 0:04:48.576 ******* 2022-01-11T13:56:32.035 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2022-01-11T13:56:32.059 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.042) 0:04:48.619 ******* 2022-01-11T13:56:32.214 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi161.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2022-01-11T13:56:32.239 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.180) 0:04:48.800 ******* 2022-01-11T13:56:32.259 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2022-01-11T13:56:32.283 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.043) 0:04:48.843 ******* 2022-01-11T13:56:32.289 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2022-01-11T13:56:32.291 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.313 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.029) 0:04:48.872 ******* 2022-01-11T13:56:32.318 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.342 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.029) 0:04:48.902 ******* 2022-01-11T13:56:32.351 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2022-01-11T13:56:32.353 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.364 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.366 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.389 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.046) 0:04:48.949 ******* 2022-01-11T13:56:32.396 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.397 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.398 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.399 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.399 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.423 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.034) 0:04:48.983 ******* 2022-01-11T13:56:32.431 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2022-01-11T13:56:32.432 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.444 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.446 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi161.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.485 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.061) 0:04:49.045 ******* 2022-01-11T13:56:32.493 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.494 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.495 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.496 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.496 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.531 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.046) 0:04:49.091 ******* 2022-01-11T13:56:32.538 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2022-01-11T13:56:32.540 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.566 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.035) 0:04:49.126 ******* 2022-01-11T13:56:32.573 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.600 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.033) 0:04:49.160 ******* 2022-01-11T13:56:32.607 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.631 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.030) 0:04:49.191 ******* 2022-01-11T13:56:32.638 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.662 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.031) 0:04:49.222 ******* 2022-01-11T13:56:32.670 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2022-01-11T13:56:32.672 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.697 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.034) 0:04:49.257 ******* 2022-01-11T13:56:32.704 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.729 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.032) 0:04:49.289 ******* 2022-01-11T13:56:32.736 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2022-01-11T13:56:32.737 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.759 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.029) 0:04:49.319 ******* 2022-01-11T13:56:32.766 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.792 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.032) 0:04:49.352 ******* 2022-01-11T13:56:32.799 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2022-01-11T13:56:32.801 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.826 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.033) 0:04:49.386 ******* 2022-01-11T13:56:32.834 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.860 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.034) 0:04:49.420 ******* 2022-01-11T13:56:32.868 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2022-01-11T13:56:32.869 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.895 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.034) 0:04:49.455 ******* 2022-01-11T13:56:32.902 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.930 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.035) 0:04:49.490 ******* 2022-01-11T13:56:32.937 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2022-01-11T13:56:32.939 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:32.969 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:32 +0000 (0:00:00.039) 0:04:49.529 ******* 2022-01-11T13:56:32.982 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:33.013 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:33 +0000 (0:00:00.042) 0:04:49.572 ******* 2022-01-11T13:56:33.020 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2022-01-11T13:56:33.021 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:33.045 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:33 +0000 (0:00:00.032) 0:04:49.605 ******* 2022-01-11T13:56:33.053 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:33.083 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:33 +0000 (0:00:00.038) 0:04:49.643 ******* 2022-01-11T13:56:33.259 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2022-01-11T13:56:33.260 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005043", "end": "2022-01-11 13:56:33.228672", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2022-01-11 13:56:33.223629", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:33.302 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004872", "end": "2022-01-11 13:56:33.268402", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2022-01-11 13:56:33.263530", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:33.439 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003034", "end": "2022-01-11 13:56:33.410297", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2022-01-11 13:56:33.407263", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:33.481 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003049", "end": "2022-01-11 13:56:33.455597", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2022-01-11 13:56:33.452548", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:33.607 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002853", "end": "2022-01-11 13:56:33.582506", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2022-01-11 13:56:33.579653", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:33.649 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002829", "end": "2022-01-11 13:56:33.624577", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2022-01-11 13:56:33.621748", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:33.777 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003319", "end": "2022-01-11 13:56:33.750851", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2022-01-11 13:56:33.747532", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:33.809 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:33 +0000 (0:00:00.726) 0:04:50.369 ******* 2022-01-11T13:56:33.828 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003203", "end": "2022-01-11 13:56:33.799054", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2022-01-11 13:56:33.795851", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:56:33.854 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:33 +0000 (0:00:00.044) 0:04:50.414 ******* 2022-01-11T13:56:33.860 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2022-01-11T13:56:33.885 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:33 +0000 (0:00:00.031) 0:04:50.445 ******* 2022-01-11T13:56:33.917 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:33 +0000 (0:00:00.030) 0:04:50.476 ******* 2022-01-11T13:56:33.922 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2022-01-11T13:56:33.924 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:33.947 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:34.025 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2022-01-11T13:56:34.042 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:34 +0000 (0:00:00.126) 0:04:50.602 ******* 2022-01-11T13:56:34.064 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:34 +0000 (0:00:00.021) 0:04:50.624 ******* 2022-01-11T13:56:34.689 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi161.front.sepia.ceph.com] 2022-01-11T13:56:34.725 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] 2022-01-11T13:56:34.751 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:34 +0000 (0:00:00.686) 0:04:51.311 ******* 2022-01-11T13:56:34.795 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:34 +0000 (0:00:00.044) 0:04:51.355 ******* 2022-01-11T13:56:34.802 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2022-01-11T13:56:34.804 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:34.826 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:34 +0000 (0:00:00.030) 0:04:51.386 ******* 2022-01-11T13:56:34.833 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:34.857 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:34 +0000 (0:00:00.031) 0:04:51.417 ******* 2022-01-11T13:56:34.864 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2022-01-11T13:56:34.866 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2022-01-11T13:56:34.889 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:34 +0000 (0:00:00.031) 0:04:51.449 ******* 2022-01-11T13:56:34.896 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2022-01-11T13:56:34.920 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:34 +0000 (0:00:00.030) 0:04:51.480 ******* 2022-01-11T13:56:34.942 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** ok: [smithi161.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2022-01-11T13:56:34.962 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:34 +0000 (0:00:00.041) 0:04:51.522 ******* 2022-01-11T13:56:34.969 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2022-01-11T13:56:34.994 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:34 +0000 (0:00:00.031) 0:04:51.554 ******* 2022-01-11T13:56:57.352 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2022-01-11T13:56:57.354 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: container-selinux-2:2.167.0-1.module_el8.5.0+911+f19012f9.noarch", "Installed: containers-common-2:1-2.module_el8.5.0+890+6b136101.noarch", "Installed: podman-3.3.1-9.module_el8.5.0+988+b1f0b741.x86_64", "Installed: podman-catatonit-3.3.1-9.module_el8.5.0+988+b1f0b741.x86_64", "Installed: podman-docker-3.3.1-9.module_el8.5.0+988+b1f0b741.noarch", "Removed: container-selinux-2:2.162.0-1.module_el8.4.0+830+8027e1c4.noarch", "Removed: containers-common-1:1.2.2-10.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-catatonit-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-docker-3.0.1-7.module_el8.4.0+830+8027e1c4.noarch"]} 2022-01-11T13:56:57.376 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:57 +0000 (0:00:22.382) 0:05:13.936 ******* 2022-01-11T13:56:57.419 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi161.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2022-01-11T13:56:57.443 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:57 +0000 (0:00:00.066) 0:05:14.003 ******* 2022-01-11T13:56:57.471 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2022-01-11T13:56:57.473 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:57.495 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:57 +0000 (0:00:00.051) 0:05:14.055 ******* 2022-01-11T13:56:57.724 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2022-01-11T13:56:57.726 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2022-01-11T13:56:57.886 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2022-01-11T13:56:57.888 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: container-selinux-2:2.167.0-1.module_el8.5.0+911+f19012f9.noarch", "Installed: containers-common-2:1-2.module_el8.5.0+890+6b136101.noarch", "Installed: podman-3.3.1-9.module_el8.5.0+988+b1f0b741.x86_64", "Installed: podman-catatonit-3.3.1-9.module_el8.5.0+988+b1f0b741.x86_64", "Installed: podman-docker-3.3.1-9.module_el8.5.0+988+b1f0b741.noarch", "Removed: container-selinux-2:2.162.0-1.module_el8.4.0+830+8027e1c4.noarch", "Removed: containers-common-1:1.2.2-10.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-catatonit-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-docker-3.0.1-7.module_el8.4.0+830+8027e1c4.noarch"]} 2022-01-11T13:56:57.910 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:57 +0000 (0:00:00.415) 0:05:14.470 ******* 2022-01-11T13:56:57.927 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2022-01-11T13:56:57.929 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2022-01-11T13:56:57.956 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:57 +0000 (0:00:00.046) 0:05:14.516 ******* 2022-01-11T13:56:57.964 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2022-01-11T13:56:57.966 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2022-01-11T13:56:57.990 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:57 +0000 (0:00:00.033) 0:05:14.550 ******* 2022-01-11T13:56:58.019 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:56:58.044 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:58 +0000 (0:00:00.054) 0:05:14.605 ******* 2022-01-11T13:56:58.281 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2022-01-11T13:56:58.283 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2022-01-11T13:56:58.439 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2022-01-11T13:56:58.441 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2022-01-11T13:56:58.461 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2022-01-11T13:56:58.464 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2022-01-11T13:56:58.485 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:58 +0000 (0:00:00.440) 0:05:15.045 ******* 2022-01-11T13:56:58.889 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2022-01-11T13:56:58.891 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2022-01-11T13:56:58.914 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:58 +0000 (0:00:00.429) 0:05:15.474 ******* 2022-01-11T13:56:58.974 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2022-01-11T13:56:59.391 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2022-01-11T13:56:59.416 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:56:59 +0000 (0:00:00.501) 0:05:15.976 ******* 2022-01-11T13:57:00.471 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2022-01-11T13:57:00.473 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:57:00.495 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:57:00 +0000 (0:00:01.079) 0:05:17.055 ******* 2022-01-11T13:57:00.975 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2022-01-11T13:57:01.001 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:57:00 +0000 (0:00:00.505) 0:05:17.560 ******* 2022-01-11T13:57:03.114 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2022-01-11T13:57:03.116 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-qeae2uoc-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-qeae2uoc-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py in 2022-01-11T13:57:03.117 INFO:teuthology.task.ansible.out:stall for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2022-01-11T13:57:03.137 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:57:03 +0000 (0:00:02.136) 0:05:19.697 ******* 2022-01-11T13:57:03.175 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2022-01-11T13:57:03.177 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:57:03.199 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:57:03 +0000 (0:00:00.062) 0:05:19.760 ******* 2022-01-11T13:57:03.227 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2022-01-11T13:57:03.228 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:57:03.250 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:57:03 +0000 (0:00:00.050) 0:05:19.810 ******* 2022-01-11T13:57:03.600 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2022-01-11T13:57:03.602 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.157723", "end": "2022-01-11 13:57:03.579734", "rc": 0, "start": "2022-01-11 13:57:03.422011", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:57:03.622 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:57:03 +0000 (0:00:00.372) 0:05:20.182 ******* 2022-01-11T13:57:03.660 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2022-01-11T13:57:03.662 INFO:teuthology.task.ansible.out:skipping: [smithi161.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:57:03.893 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2022-01-11T13:57:03.894 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-xis16j7g-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-xis16j7g-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py in 2022-01-11T13:57:03.896 INFO:teuthology.task.ansible.out:stall for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2022-01-11T13:57:03.914 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:57:03 +0000 (0:00:00.292) 0:05:20.475 ******* 2022-01-11T13:57:03.946 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2022-01-11T13:57:03.952 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:57:03.969 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:57:03 +0000 (0:00:00.054) 0:05:20.529 ******* 2022-01-11T13:57:03.996 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2022-01-11T13:57:03.998 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:57:04.020 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:57:04 +0000 (0:00:00.051) 0:05:20.580 ******* 2022-01-11T13:57:04.364 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2022-01-11T13:57:04.366 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.153624", "end": "2022-01-11 13:57:04.348519", "rc": 0, "start": "2022-01-11 13:57:04.194895", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2022-01-11T13:57:04.387 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:57:04 +0000 (0:00:00.366) 0:05:20.947 ******* 2022-01-11T13:57:04.425 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2022-01-11T13:57:04.506 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2022-01-11T13:57:04.507 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2022-01-11T13:57:04.509 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2022-01-11T13:57:04.510 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2022-01-11T13:57:04.523 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2022-01-11T13:57:04.525 INFO:teuthology.task.ansible.out:Tuesday 11 January 2022 13:57:04 +0000 (0:00:00.136) 0:05:21.083 ******* 2022-01-11T13:57:05.173 INFO:teuthology.task.ansible.out:ok: [smithi161.front.sepia.ceph.com] 2022-01-11T13:57:05.204 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] 2022-01-11T13:57:05.221 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** Tuesday 11 January 2022 13:57:05 +0000 (0:00:00.697) 0:05:21.781 ******* 2022-01-11T13:57:05.415 INFO:teuthology.task.ansible.out:changed: [smithi161.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2022-01-11T13:57:05.436 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2022-01-11T13:57:05.439 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi161.front.sepia.ceph.com : ok=135 changed=44 unreachable=0 failed=0 skipped=184 rescued=0 ignored=3 2022-01-11T13:57:05.441 INFO:teuthology.task.ansible.out:smithi172.front.sepia.ceph.com : ok=135 changed=44 unreachable=0 failed=0 skipped=184 rescued=0 ignored=3 Tuesday 11 January 2022 13:57:05 +0000 (0:00:00.218) 0:05:21.999 ******* =============================================================================== Create all admin users with sudo access. ------------------------------- 41.21s common : nrpe - Ensure SELinux policy is up to date -------------------- 38.08s users : Update authorized_keys using the keys repo --------------------- 32.42s container-host : Install container packages ---------------------------- 22.38s testnode : Ensure ceph packages are not present. ----------------------- 20.47s Remove revoked users --------------------------------------------------- 18.58s common : nrpe - Remove SELinux policy package -------------------------- 15.22s common : nrpe - Load SELinux policy package ---------------------------- 15.19s testnode : Install packages --------------------------------------------- 8.59s testnode : Zap all non-root disks --------------------------------------- 7.52s testnode : Install GPG -------------------------------------------------- 4.69s common : Install Kerberos Packages (RedHat) ----------------------------- 3.04s testnode : Install GPG keys --------------------------------------------- 2.79s testnode : Upgrade packages --------------------------------------------- 2.60s testnode : Rebuild rpmdb ------------------------------------------------ 2.51s testnode : Ensure ceph dependency packages are not present. ------------- 2.37s container-host : Install registries-conf-ctl ---------------------------- 2.14s users : Update authorized_keys for each user with literal keys ---------- 1.91s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.75s testnode : Configure local mirrorlists ---------------------------------- 1.73s 2022-01-11T13:57:05.881 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi161.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2022-01-11T13:57:05.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2022-01-11T13:57:05.884 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-11T13:57:05.956 DEBUG:teuthology.orchestra.run.smithi161:> true 2022-01-11T13:57:06.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2022-01-11T13:57:06.044 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-01-11T13:57:06.116 DEBUG:teuthology.orchestra.run.smithi172:> true 2022-01-11T13:57:06.202 INFO:teuthology.run_tasks:Running task clock... 2022-01-11T13:57:06.219 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-01-11T13:57:06.219 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-01-11T13:57:06.221 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-01-11T13:57:06.224 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-01-11T13:57:06.224 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-01-11T13:57:06.253 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-01-11T13:57:06.268 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-01-11T13:57:06.272 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-01-11T13:57:06.287 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-01-11T13:57:06.301 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: ntpd: command not found 2022-01-11T13:57:06.311 INFO:teuthology.orchestra.run.smithi161.stdout:506 Cannot talk to daemon 2022-01-11T13:57:06.322 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2022-01-11T13:57:06.324 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-01-11T13:57:06.332 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2022-01-11T13:57:06.338 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-01-11T13:57:06.345 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-01-11T13:57:06.360 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-01-11T13:57:06.413 INFO:teuthology.orchestra.run.smithi161.stderr:bash: ntpq: command not found 2022-01-11T13:57:06.415 INFO:teuthology.orchestra.run.smithi161.stdout:210 Number of sources = 4 2022-01-11T13:57:06.416 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-01-11T13:57:06.416 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2022-01-11T13:57:06.416 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-11T13:57:06.416 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-11T13:57:06.416 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-11T13:57:06.417 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-11T13:57:06.427 INFO:teuthology.orchestra.run.smithi172.stderr:bash: ntpq: command not found 2022-01-11T13:57:06.429 INFO:teuthology.orchestra.run.smithi172.stdout:210 Number of sources = 4 2022-01-11T13:57:06.430 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-01-11T13:57:06.430 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2022-01-11T13:57:06.430 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-11T13:57:06.430 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-11T13:57:06.431 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-11T13:57:06.431 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-01-11T13:57:06.432 INFO:teuthology.run_tasks:Running task install... 2022-01-11T13:57:06.444 DEBUG:teuthology.task.install:project ceph 2022-01-11T13:57:06.445 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '17f142526478a11e8f62d3933be552d546c4fd19'}} 2022-01-11T13:57:06.445 DEBUG:teuthology.task.install:config {'sha1': '17f142526478a11e8f62d3933be552d546c4fd19'} 2022-01-11T13:57:06.445 INFO:teuthology.task.install:Using flavor: default 2022-01-11T13:57:06.452 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-01-11T13:57:06.453 INFO:teuthology.task.install:extra packages: [] 2022-01-11T13:57:06.453 DEBUG:teuthology.orchestra.run.smithi161:> sudo lsb_release -is 2022-01-11T13:57:06.489 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS 2022-01-11T13:57:06.490 DEBUG:teuthology.misc:System to be installed: CentOS 2022-01-11T13:57:06.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo lsb_release -is 2022-01-11T13:57:06.491 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '17f142526478a11e8f62d3933be552d546c4fd19', 'tag': None, 'wait_for_package': False} 2022-01-11T13:57:06.492 DEBUG:teuthology.orchestra.run.smithi161:> sudo lsb_release -is 2022-01-11T13:57:06.525 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS 2022-01-11T13:57:06.526 DEBUG:teuthology.misc:System to be installed: CentOS 2022-01-11T13:57:06.527 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '17f142526478a11e8f62d3933be552d546c4fd19', 'tag': None, 'wait_for_package': False} 2022-01-11T13:57:06.527 DEBUG:teuthology.orchestra.run.smithi172:> sudo lsb_release -is 2022-01-11T13:57:06.562 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS 2022-01-11T13:57:06.563 DEBUG:teuthology.misc:System to be installed: CentOS 2022-01-11T13:57:06.564 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=17f142526478a11e8f62d3933be552d546c4fd19 2022-01-11T13:57:06.600 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS 2022-01-11T13:57:06.601 DEBUG:teuthology.misc:System to be installed: CentOS 2022-01-11T13:57:06.602 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=17f142526478a11e8f62d3933be552d546c4fd19 2022-01-11T13:57:06.784 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-dang-testing/17f142526478a11e8f62d3933be552d546c4fd19/centos/8/flavors/default/ 2022-01-11T13:57:06.785 INFO:teuthology.task.install.rpm:Package version is 17.0.0-10004.g17f14252 2022-01-11T13:57:06.814 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-dang-testing/17f142526478a11e8f62d3933be552d546c4fd19/centos/8/flavors/default/ 2022-01-11T13:57:06.814 INFO:teuthology.task.install.rpm:Package version is 17.0.0-10004.g17f14252 2022-01-11T13:57:06.955 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-dang-testing/17f142526478a11e8f62d3933be552d546c4fd19/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-dang-testing/17f142526478a11e8f62d3933be552d546c4fd19/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-dang-testing/17f142526478a11e8f62d3933be552d546c4fd19/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-01-11T13:57:06.956 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:57:06.956 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-01-11T13:57:06.973 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-dang-testing/17f142526478a11e8f62d3933be552d546c4fd19/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-dang-testing/17f142526478a11e8f62d3933be552d546c4fd19/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-dang-testing/17f142526478a11e8f62d3933be552d546c4fd19/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-01-11T13:57:06.973 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T13:57:06.974 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-01-11T13:57:06.981 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-01-11T13:57:06.981 DEBUG:teuthology.orchestra.run.smithi161:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/17f142526478a11e8f62d3933be552d546c4fd19/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-01-11T13:57:06.998 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2022-01-11T13:57:06.998 DEBUG:teuthology.orchestra.run.smithi172:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/17f142526478a11e8f62d3933be552d546c4fd19/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-01-11T13:57:07.045 DEBUG:teuthology.orchestra.run.smithi161:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-01-11T13:57:07.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-01-11T13:57:07.120 DEBUG:teuthology.orchestra.run.smithi161:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-01-11T13:57:07.137 DEBUG:teuthology.orchestra.run.smithi172:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-01-11T13:57:07.183 INFO:teuthology.orchestra.run.smithi161.stdout:check_obsoletes = 1 2022-01-11T13:57:07.185 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum clean all 2022-01-11T13:57:07.201 INFO:teuthology.orchestra.run.smithi172.stdout:check_obsoletes = 1 2022-01-11T13:57:07.202 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum clean all 2022-01-11T13:57:07.423 INFO:teuthology.orchestra.run.smithi161.stdout:93 files removed 2022-01-11T13:57:07.441 INFO:teuthology.orchestra.run.smithi172.stdout:93 files removed 2022-01-11T13:57:07.443 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-radosgw 2022-01-11T13:57:07.462 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-radosgw 2022-01-11T13:57:08.151 INFO:teuthology.orchestra.run.smithi172.stdout:ceph packages for x86_64 212 kB/s | 75 kB 00:00 2022-01-11T13:57:08.168 INFO:teuthology.orchestra.run.smithi161.stdout:ceph packages for x86_64 191 kB/s | 75 kB 00:00 2022-01-11T13:57:08.524 INFO:teuthology.orchestra.run.smithi172.stdout:ceph noarch packages 27 kB/s | 9.1 kB 00:00 2022-01-11T13:57:08.529 INFO:teuthology.orchestra.run.smithi161.stdout:ceph noarch packages 28 kB/s | 9.1 kB 00:00 2022-01-11T13:57:08.857 INFO:teuthology.orchestra.run.smithi161.stdout:ceph source packages 4.7 kB/s | 1.4 kB 00:00 2022-01-11T13:57:08.859 INFO:teuthology.orchestra.run.smithi172.stdout:ceph source packages 4.6 kB/s | 1.4 kB 00:00 2022-01-11T13:57:09.092 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-8 - AppStream 40 MB/s | 8.4 MB 00:00 2022-01-11T13:57:09.122 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-8 - AppStream 36 MB/s | 8.4 MB 00:00 2022-01-11T13:57:10.517 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-8 - Base 40 MB/s | 4.6 MB 00:00 2022-01-11T13:57:10.592 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-8 - Base 29 MB/s | 4.6 MB 00:00 2022-01-11T13:57:11.216 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-8 - Extras 155 kB/s | 10 kB 00:00 2022-01-11T13:57:11.291 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-8 - Extras 164 kB/s | 10 kB 00:00 2022-01-11T13:57:13.110 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Linux 8 - AppStream 4.8 MB/s | 8.4 MB 00:01 2022-01-11T13:57:13.132 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Linux 8 - AppStream 4.5 MB/s | 8.4 MB 00:01 2022-01-11T13:57:14.896 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Linux 8 - BaseOS 9.9 MB/s | 4.6 MB 00:00 2022-01-11T13:57:14.940 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Linux 8 - BaseOS 12 MB/s | 4.6 MB 00:00 2022-01-11T13:57:15.663 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Linux 8 - Extras 84 kB/s | 10 kB 00:00 2022-01-11T13:57:15.812 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-8 - PowerTools 26 MB/s | 2.3 MB 00:00 2022-01-11T13:57:15.906 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Linux 8 - Extras 32 kB/s | 10 kB 00:00 2022-01-11T13:57:16.052 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-8 - PowerTools 27 MB/s | 2.3 MB 00:00 2022-01-11T13:57:18.677 INFO:teuthology.orchestra.run.smithi161.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2022-01-11T13:57:18.692 INFO:teuthology.orchestra.run.smithi172.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2022-01-11T13:57:20.004 INFO:teuthology.orchestra.run.smithi161.stdout:Apache Arrow 146 kB/s | 184 kB 00:01 2022-01-11T13:57:20.023 INFO:teuthology.orchestra.run.smithi172.stdout:Apache Arrow 146 kB/s | 184 kB 00:01 2022-01-11T13:57:20.917 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 14 MB/s | 11 MB 00:00 2022-01-11T13:57:20.921 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 14 MB/s | 11 MB 00:00 2022-01-11T13:57:23.276 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2022-01-11T13:57:23.323 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2022-01-11T13:57:25.516 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:57:25.519 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:57:25.519 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2022-01-11T13:57:25.520 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:57:25.520 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:57:25.520 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw x86_64 2:17.0.0-10004.g17f14252.el8 ceph 9.1 M 2022-01-11T13:57:25.521 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrading: 2022-01-11T13:57:25.521 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 x86_64 2:17.0.0-10004.g17f14252.el8 ceph 3.4 M 2022-01-11T13:57:25.521 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1 x86_64 2:17.0.0-10004.g17f14252.el8 ceph 3.9 M 2022-01-11T13:57:25.521 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2022-01-11T13:57:25.522 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base x86_64 2:17.0.0-10004.g17f14252.el8 ceph 5.2 M 2022-01-11T13:57:25.522 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common x86_64 2:17.0.0-10004.g17f14252.el8 ceph 20 M 2022-01-11T13:57:25.522 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux x86_64 2:17.0.0-10004.g17f14252.el8 ceph 24 k 2022-01-11T13:57:25.522 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-01-11T13:57:25.523 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2 x86_64 2:17.0.0-10004.g17f14252.el8 ceph 787 k 2022-01-11T13:57:25.523 INFO:teuthology.orchestra.run.smithi172.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-01-11T13:57:25.523 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2022-01-11T13:57:25.523 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 x86_64 2:17.0.0-10004.g17f14252.el8 ceph 483 k 2022-01-11T13:57:25.524 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2022-01-11T13:57:25.524 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2 x86_64 2:17.0.0-10004.g17f14252.el8 ceph 3.9 M 2022-01-11T13:57:25.524 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-01-11T13:57:25.525 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-01-11T13:57:25.525 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse 2022-01-11T13:57:25.525 INFO:teuthology.orchestra.run.smithi172.stdout: x86_64 2:17.0.0-10004.g17f14252.el8 ceph 45 k 2022-01-11T13:57:25.525 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common x86_64 2:17.0.0-10004.g17f14252.el8 ceph 103 k 2022-01-11T13:57:25.526 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs x86_64 2:17.0.0-10004.g17f14252.el8 ceph 177 k 2022-01-11T13:57:25.526 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-01-11T13:57:25.526 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados x86_64 2:17.0.0-10004.g17f14252.el8 ceph 367 k 2022-01-11T13:57:25.526 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd x86_64 2:17.0.0-10004.g17f14252.el8 ceph 345 k 2022-01-11T13:57:25.527 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw x86_64 2:17.0.0-10004.g17f14252.el8 ceph 109 k 2022-01-11T13:57:25.527 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:57:25.527 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:57:25.527 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:57:25.527 INFO:teuthology.orchestra.run.smithi172.stdout:Install 20 Packages 2022-01-11T13:57:25.528 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrade 2 Packages 2022-01-11T13:57:25.528 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:57:25.529 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 49 M 2022-01-11T13:57:25.529 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:57:25.562 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:57:25.564 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:57:25.565 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2022-01-11T13:57:25.565 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:57:25.565 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:57:25.565 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw x86_64 2:17.0.0-10004.g17f14252.el8 ceph 9.1 M 2022-01-11T13:57:25.566 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrading: 2022-01-11T13:57:25.566 INFO:teuthology.orchestra.run.smithi161.stdout: librados2 x86_64 2:17.0.0-10004.g17f14252.el8 ceph 3.4 M 2022-01-11T13:57:25.566 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1 x86_64 2:17.0.0-10004.g17f14252.el8 ceph 3.9 M 2022-01-11T13:57:25.566 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2022-01-11T13:57:25.567 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base x86_64 2:17.0.0-10004.g17f14252.el8 ceph 5.2 M 2022-01-11T13:57:25.567 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common x86_64 2:17.0.0-10004.g17f14252.el8 ceph 20 M 2022-01-11T13:57:25.567 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux x86_64 2:17.0.0-10004.g17f14252.el8 ceph 24 k 2022-01-11T13:57:25.567 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-01-11T13:57:25.568 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2 x86_64 2:17.0.0-10004.g17f14252.el8 ceph 787 k 2022-01-11T13:57:25.568 INFO:teuthology.orchestra.run.smithi161.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-01-11T13:57:25.568 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2022-01-11T13:57:25.568 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1 x86_64 2:17.0.0-10004.g17f14252.el8 ceph 483 k 2022-01-11T13:57:25.569 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2022-01-11T13:57:25.569 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2 x86_64 2:17.0.0-10004.g17f14252.el8 ceph 3.9 M 2022-01-11T13:57:25.569 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-01-11T13:57:25.569 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-01-11T13:57:25.570 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse 2022-01-11T13:57:25.570 INFO:teuthology.orchestra.run.smithi161.stdout: x86_64 2:17.0.0-10004.g17f14252.el8 ceph 45 k 2022-01-11T13:57:25.570 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common x86_64 2:17.0.0-10004.g17f14252.el8 ceph 103 k 2022-01-11T13:57:25.571 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs x86_64 2:17.0.0-10004.g17f14252.el8 ceph 177 k 2022-01-11T13:57:25.571 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-01-11T13:57:25.571 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados x86_64 2:17.0.0-10004.g17f14252.el8 ceph 367 k 2022-01-11T13:57:25.571 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd x86_64 2:17.0.0-10004.g17f14252.el8 ceph 345 k 2022-01-11T13:57:25.572 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw x86_64 2:17.0.0-10004.g17f14252.el8 ceph 109 k 2022-01-11T13:57:25.572 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:57:25.572 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:57:25.573 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:57:25.573 INFO:teuthology.orchestra.run.smithi161.stdout:Install 20 Packages 2022-01-11T13:57:25.573 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrade 2 Packages 2022-01-11T13:57:25.574 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:57:25.575 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 49 M 2022-01-11T13:57:25.575 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:57:26.944 INFO:teuthology.orchestra.run.smithi161.stdout:(1/22): ceph-base-17.0.0-10004.g17f14252.el8.x8 3.8 MB/s | 5.2 MB 00:01 2022-01-11T13:57:26.965 INFO:teuthology.orchestra.run.smithi172.stdout:(1/22): ceph-base-17.0.0-10004.g17f14252.el8.x8 3.6 MB/s | 5.2 MB 00:01 2022-01-11T13:57:27.003 INFO:teuthology.orchestra.run.smithi161.stdout:(2/22): ceph-selinux-17.0.0-10004.g17f14252.el8 412 kB/s | 24 kB 00:00 2022-01-11T13:57:27.029 INFO:teuthology.orchestra.run.smithi172.stdout:(2/22): ceph-selinux-17.0.0-10004.g17f14252.el8 387 kB/s | 24 kB 00:00 2022-01-11T13:57:27.168 INFO:teuthology.orchestra.run.smithi161.stdout:(3/22): libcephfs2-17.0.0-10004.g17f14252.el8.x 4.7 MB/s | 787 kB 00:00 2022-01-11T13:57:27.214 INFO:teuthology.orchestra.run.smithi172.stdout:(3/22): libcephfs2-17.0.0-10004.g17f14252.el8.x 4.1 MB/s | 787 kB 00:00 2022-01-11T13:57:27.343 INFO:teuthology.orchestra.run.smithi161.stdout:(4/22): libradosstriper1-17.0.0-10004.g17f14252 2.7 MB/s | 483 kB 00:00 2022-01-11T13:57:27.513 INFO:teuthology.orchestra.run.smithi172.stdout:(4/22): ceph-radosgw-17.0.0-10004.g17f14252.el8 4.6 MB/s | 9.1 MB 00:01 2022-01-11T13:57:27.514 INFO:teuthology.orchestra.run.smithi172.stdout:(5/22): libradosstriper1-17.0.0-10004.g17f14252 1.6 MB/s | 483 kB 00:00 2022-01-11T13:57:27.580 INFO:teuthology.orchestra.run.smithi161.stdout:(5/22): ceph-radosgw-17.0.0-10004.g17f14252.el8 4.5 MB/s | 9.1 MB 00:02 2022-01-11T13:57:27.581 INFO:teuthology.orchestra.run.smithi172.stdout:(6/22): python3-ceph-argparse-17.0.0-10004.g17f 670 kB/s | 45 kB 00:00 2022-01-11T13:57:27.637 INFO:teuthology.orchestra.run.smithi161.stdout:(6/22): python3-ceph-argparse-17.0.0-10004.g17f 785 kB/s | 45 kB 00:00 2022-01-11T13:57:27.647 INFO:teuthology.orchestra.run.smithi172.stdout:(7/22): python3-ceph-common-17.0.0-10004.g17f14 1.6 MB/s | 103 kB 00:00 2022-01-11T13:57:27.701 INFO:teuthology.orchestra.run.smithi161.stdout:(7/22): python3-ceph-common-17.0.0-10004.g17f14 1.6 MB/s | 103 kB 00:00 2022-01-11T13:57:27.714 INFO:teuthology.orchestra.run.smithi172.stdout:(8/22): python3-cephfs-17.0.0-10004.g17f14252.e 2.5 MB/s | 177 kB 00:00 2022-01-11T13:57:27.771 INFO:teuthology.orchestra.run.smithi161.stdout:(8/22): python3-cephfs-17.0.0-10004.g17f14252.e 2.5 MB/s | 177 kB 00:00 2022-01-11T13:57:27.866 INFO:teuthology.orchestra.run.smithi172.stdout:(9/22): python3-rados-17.0.0-10004.g17f14252.el 2.4 MB/s | 367 kB 00:00 2022-01-11T13:57:27.876 INFO:teuthology.orchestra.run.smithi161.stdout:(9/22): librgw2-17.0.0-10004.g17f14252.el8.x86_ 7.4 MB/s | 3.9 MB 00:00 2022-01-11T13:57:27.906 INFO:teuthology.orchestra.run.smithi161.stdout:(10/22): python3-rados-17.0.0-10004.g17f14252.e 2.7 MB/s | 367 kB 00:00 2022-01-11T13:57:28.048 INFO:teuthology.orchestra.run.smithi172.stdout:(10/22): python3-rbd-17.0.0-10004.g17f14252.el8 3.1 MB/s | 345 kB 00:00 2022-01-11T13:57:28.048 INFO:teuthology.orchestra.run.smithi172.stdout:(11/22): python3-rgw-17.0.0-10004.g17f14252.el8 2.3 MB/s | 109 kB 00:00 2022-01-11T13:57:28.049 INFO:teuthology.orchestra.run.smithi161.stdout:(11/22): python3-rbd-17.0.0-10004.g17f14252.el8 5.3 MB/s | 345 kB 00:00 2022-01-11T13:57:28.050 INFO:teuthology.orchestra.run.smithi161.stdout:(12/22): python3-rgw-17.0.0-10004.g17f14252.el8 1.6 MB/s | 109 kB 00:00 2022-01-11T13:57:28.050 INFO:teuthology.orchestra.run.smithi161.stdout:(13/22): librdkafka-0.11.4-1.el8.x86_64.rpm 8.9 MB/s | 353 kB 00:00 2022-01-11T13:57:28.050 INFO:teuthology.orchestra.run.smithi161.stdout:(14/22): python3-prettytable-0.7.2-14.el8.noarc 28 MB/s | 44 kB 00:00 2022-01-11T13:57:28.051 INFO:teuthology.orchestra.run.smithi161.stdout:(15/22): librabbitmq-0.9.0-3.el8.x86_64.rpm 29 MB/s | 47 kB 00:00 2022-01-11T13:57:28.051 INFO:teuthology.orchestra.run.smithi161.stdout:(16/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.9 MB/s | 259 kB 00:00 2022-01-11T13:57:28.060 INFO:teuthology.orchestra.run.smithi172.stdout:(12/22): librdkafka-0.11.4-1.el8.x86_64.rpm 9.4 MB/s | 353 kB 00:00 2022-01-11T13:57:28.064 INFO:teuthology.orchestra.run.smithi172.stdout:(13/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 69 MB/s | 259 kB 00:00 2022-01-11T13:57:28.065 INFO:teuthology.orchestra.run.smithi172.stdout:(14/22): python3-prettytable-0.7.2-14.el8.noarc 34 MB/s | 44 kB 00:00 2022-01-11T13:57:28.067 INFO:teuthology.orchestra.run.smithi172.stdout:(15/22): librabbitmq-0.9.0-3.el8.x86_64.rpm 34 MB/s | 47 kB 00:00 2022-01-11T13:57:28.130 INFO:teuthology.orchestra.run.smithi172.stdout:(16/22): librgw2-17.0.0-10004.g17f14252.el8.x86 6.4 MB/s | 3.9 MB 00:00 2022-01-11T13:57:28.134 INFO:teuthology.orchestra.run.smithi161.stdout:(17/22): liboath-2.6.2-3.el8.x86_64.rpm 453 kB/s | 59 kB 00:00 2022-01-11T13:57:28.191 INFO:teuthology.orchestra.run.smithi161.stdout:(18/22): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2022-01-11T13:57:28.211 INFO:teuthology.orchestra.run.smithi161.stdout:(19/22): libunwind-1.3.1-3.el8.x86_64.rpm 967 kB/s | 75 kB 00:00 2022-01-11T13:57:28.238 INFO:teuthology.orchestra.run.smithi172.stdout:(17/22): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2022-01-11T13:57:28.240 INFO:teuthology.orchestra.run.smithi172.stdout:(18/22): liboath-2.6.2-3.el8.x86_64.rpm 534 kB/s | 59 kB 00:00 2022-01-11T13:57:28.267 INFO:teuthology.orchestra.run.smithi172.stdout:(19/22): libunwind-1.3.1-3.el8.x86_64.rpm 2.5 MB/s | 75 kB 00:00 2022-01-11T13:57:28.698 INFO:teuthology.orchestra.run.smithi161.stdout:(20/22): librados2-17.0.0-10004.g17f14252.el8.x 6.8 MB/s | 3.4 MB 00:00 2022-01-11T13:57:29.019 INFO:teuthology.orchestra.run.smithi161.stdout:(21/22): librbd1-17.0.0-10004.g17f14252.el8.x86 4.8 MB/s | 3.9 MB 00:00 2022-01-11T13:57:29.032 INFO:teuthology.orchestra.run.smithi172.stdout:(20/22): librbd1-17.0.0-10004.g17f14252.el8.x86 5.1 MB/s | 3.9 MB 00:00 2022-01-11T13:57:29.148 INFO:teuthology.orchestra.run.smithi172.stdout:(21/22): ceph-common-17.0.0-10004.g17f14252.el8 5.6 MB/s | 20 MB 00:03 2022-01-11T13:57:29.159 INFO:teuthology.orchestra.run.smithi172.stdout:(22/22): librados2-17.0.0-10004.g17f14252.el8.x 3.7 MB/s | 3.4 MB 00:00 2022-01-11T13:57:29.161 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:57:29.162 INFO:teuthology.orchestra.run.smithi172.stdout:Total 14 MB/s | 49 MB 00:03 2022-01-11T13:57:29.169 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:57:29.212 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:57:29.212 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:57:29.315 INFO:teuthology.orchestra.run.smithi161.stdout:(22/22): ceph-common-17.0.0-10004.g17f14252.el8 5.4 MB/s | 20 MB 00:03 2022-01-11T13:57:29.319 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:57:29.320 INFO:teuthology.orchestra.run.smithi161.stdout:Total 13 MB/s | 49 MB 00:03 2022-01-11T13:57:29.326 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:57:29.370 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:57:29.370 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:57:29.593 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:57:29.596 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:57:29.721 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:57:29.724 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:57:30.651 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:57:30.722 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:57:30.850 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2022-01-11T13:57:30.887 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2022-01-11T13:57:31.047 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2022-01-11T13:57:31.050 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2022-01-11T13:57:31.436 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2022-01-11T13:57:31.443 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2022-01-11T13:57:31.600 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librados2-2:17.0.0-10004.g17f14252.el8.x86_64 3/24 2022-01-11T13:57:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librados2-2:17.0.0-10004.g17f14252.el8.x86_64 3/24 2022-01-11T13:57:31.703 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:17.0.0-10004.g17f14252.el8.x86_64 3/24 2022-01-11T13:57:31.704 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:17.0.0-10004.g17f14252.el8.x86_64 3/24 2022-01-11T13:57:31.900 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rados-2:17.0.0-10004.g17f14252.el8.x86_64 4/24 2022-01-11T13:57:31.946 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rados-2:17.0.0-10004.g17f14252.el8.x86_64 4/24 2022-01-11T13:57:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs2-2:17.0.0-10004.g17f14252.el8.x86_64 5/24 2022-01-11T13:57:32.080 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs2-2:17.0.0-10004.g17f14252.el8.x86_64 5/24 2022-01-11T13:57:32.483 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs2-2:17.0.0-10004.g17f14252.el8.x86_64 5/24 2022-01-11T13:57:32.533 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs2-2:17.0.0-10004.g17f14252.el8.x86_64 5/24 2022-01-11T13:57:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librbd1-2:17.0.0-10004.g17f14252.el8.x86_64 6/24 2022-01-11T13:57:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:17.0.0-10004.g17f14252.el8.x86_64 6/24 2022-01-11T13:57:32.713 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librbd1-2:17.0.0-10004.g17f14252.el8.x86_64 6/24 2022-01-11T13:57:32.778 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:17.0.0-10004.g17f14252.el8.x86_64 6/24 2022-01-11T13:57:32.840 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/24 2022-01-11T13:57:32.912 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/24 2022-01-11T13:57:33.024 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/24 2022-01-11T13:57:33.080 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/24 2022-01-11T13:57:33.430 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/24 2022-01-11T13:57:33.500 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/24 2022-01-11T13:57:33.591 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librgw2-2:17.0.0-10004.g17f14252.el8.x86_64 9/24 2022-01-11T13:57:33.645 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librgw2-2:17.0.0-10004.g17f14252.el8.x86_64 9/24 2022-01-11T13:57:33.663 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librgw2-2:17.0.0-10004.g17f14252.el8.x86_64 9/24 2022-01-11T13:57:33.732 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librgw2-2:17.0.0-10004.g17f14252.el8.x86_64 9/24 2022-01-11T13:57:33.799 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-argparse-2:17.0.0-10004.g17f14252.el8 10/24 2022-01-11T13:57:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-argparse-2:17.0.0-10004.g17f14252.el8 10/24 2022-01-11T13:57:33.952 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cephfs-2:17.0.0-10004.g17f14252.el8.x86_64 11/24 2022-01-11T13:57:34.075 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cephfs-2:17.0.0-10004.g17f14252.el8.x86_64 11/24 2022-01-11T13:57:34.149 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rgw-2:17.0.0-10004.g17f14252.el8.x86_64 12/24 2022-01-11T13:57:34.322 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rgw-2:17.0.0-10004.g17f14252.el8.x86_64 12/24 2022-01-11T13:57:34.367 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rbd-2:17.0.0-10004.g17f14252.el8.x86_64 13/24 2022-01-11T13:57:34.507 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libradosstriper1-2:17.0.0-10004.g17f14252.el8.x86_ 14/24 2022-01-11T13:57:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rbd-2:17.0.0-10004.g17f14252.el8.x86_64 13/24 2022-01-11T13:57:34.575 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libradosstriper1-2:17.0.0-10004.g17f14252.el8.x86_ 14/24 2022-01-11T13:57:34.654 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradosstriper1-2:17.0.0-10004.g17f14252.el8.x86_ 14/24 2022-01-11T13:57:34.753 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libradosstriper1-2:17.0.0-10004.g17f14252.el8.x86_ 14/24 2022-01-11T13:57:34.774 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2022-01-11T13:57:34.914 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2022-01-11T13:57:34.982 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2022-01-11T13:57:35.138 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2022-01-11T13:57:35.158 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2022-01-11T13:57:35.332 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-common-2:17.0.0-10004.g17f14252.el8.x 18/24 2022-01-11T13:57:35.339 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2022-01-11T13:57:35.547 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-common-2:17.0.0-10004.g17f14252.el8.x 18/24 2022-01-11T13:57:39.570 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:17.0.0-10004.g17f14252.el8.x86_64 19/24 2022-01-11T13:57:39.621 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:17.0.0-10004.g17f14252.el8.x86_64 19/24 2022-01-11T13:57:39.756 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-common-2:17.0.0-10004.g17f14252.el8.x86_64 19/24 2022-01-11T13:57:39.837 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-common-2:17.0.0-10004.g17f14252.el8.x86_64 19/24 2022-01-11T13:57:40.400 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:17.0.0-10004.g17f14252.el8.x86_64 19/24 2022-01-11T13:57:40.444 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:17.0.0-10004.g17f14252.el8.x86_64 19/24 2022-01-11T13:57:40.653 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-base-2:17.0.0-10004.g17f14252.el8.x86_64 20/24 2022-01-11T13:57:40.700 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-base-2:17.0.0-10004.g17f14252.el8.x86_64 20/24 2022-01-11T13:57:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-base-2:17.0.0-10004.g17f14252.el8.x86_64 20/24 2022-01-11T13:57:40.803 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-base-2:17.0.0-10004.g17f14252.el8.x86_64 20/24 2022-01-11T13:57:40.873 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-selinux-2:17.0.0-10004.g17f14252.el8.x86_64 21/24 2022-01-11T13:57:40.929 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-selinux-2:17.0.0-10004.g17f14252.el8.x86_64 21/24 2022-01-11T13:57:56.375 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-selinux-2:17.0.0-10004.g17f14252.el8.x86_64 21/24 2022-01-11T13:57:56.376 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /sys 2022-01-11T13:57:56.376 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /proc 2022-01-11T13:57:56.377 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /dev 2022-01-11T13:57:56.377 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /run 2022-01-11T13:57:56.377 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /mnt 2022-01-11T13:57:56.377 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /var/tmp 2022-01-11T13:57:56.377 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /home 2022-01-11T13:57:56.378 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /tmp 2022-01-11T13:57:56.378 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /dev 2022-01-11T13:57:56.378 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:57:56.389 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-selinux-2:17.0.0-10004.g17f14252.el8.x86_64 21/24 2022-01-11T13:57:56.390 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /sys 2022-01-11T13:57:56.390 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /proc 2022-01-11T13:57:56.390 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /dev 2022-01-11T13:57:56.390 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /run 2022-01-11T13:57:56.391 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /mnt 2022-01-11T13:57:56.391 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /var/tmp 2022-01-11T13:57:56.391 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /home 2022-01-11T13:57:56.391 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /tmp 2022-01-11T13:57:56.392 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /dev 2022-01-11T13:57:56.392 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:57:57.514 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-radosgw-2:17.0.0-10004.g17f14252.el8.x86_64 22/24 2022-01-11T13:57:57.516 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-radosgw-2:17.0.0-10004.g17f14252.el8.x86_64 22/24 2022-01-11T13:57:57.592 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-10004.g17f14252.el8.x86_64 22/24 2022-01-11T13:57:57.592 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2022-01-11T13:57:57.602 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-10004.g17f14252.el8.x86_64 22/24 2022-01-11T13:57:57.602 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2022-01-11T13:57:57.822 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2022-01-11T13:57:57.823 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2022-01-11T13:57:57.844 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2022-01-11T13:57:57.845 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2022-01-11T13:57:58.517 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2022-01-11T13:57:58.518 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-base-2:17.0.0-10004.g17f14252.el8.x86_64 1/24 2022-01-11T13:57:58.518 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-common-2:17.0.0-10004.g17f14252.el8.x86_64 2/24 2022-01-11T13:57:58.518 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-radosgw-2:17.0.0-10004.g17f14252.el8.x86_64 3/24 2022-01-11T13:57:58.518 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-selinux-2:17.0.0-10004.g17f14252.el8.x86_64 4/24 2022-01-11T13:57:58.518 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs2-2:17.0.0-10004.g17f14252.el8.x86_64 5/24 2022-01-11T13:57:58.519 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libradosstriper1-2:17.0.0-10004.g17f14252.el8.x86_ 6/24 2022-01-11T13:57:58.519 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librgw2-2:17.0.0-10004.g17f14252.el8.x86_64 7/24 2022-01-11T13:57:58.519 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-argparse-2:17.0.0-10004.g17f14252.el8 8/24 2022-01-11T13:57:58.519 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-common-2:17.0.0-10004.g17f14252.el8.x 9/24 2022-01-11T13:57:58.520 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cephfs-2:17.0.0-10004.g17f14252.el8.x86_64 10/24 2022-01-11T13:57:58.520 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rados-2:17.0.0-10004.g17f14252.el8.x86_64 11/24 2022-01-11T13:57:58.520 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rbd-2:17.0.0-10004.g17f14252.el8.x86_64 12/24 2022-01-11T13:57:58.520 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rgw-2:17.0.0-10004.g17f14252.el8.x86_64 13/24 2022-01-11T13:57:58.521 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/24 2022-01-11T13:57:58.521 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/24 2022-01-11T13:57:58.521 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/24 2022-01-11T13:57:58.521 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/24 2022-01-11T13:57:58.521 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/24 2022-01-11T13:57:58.522 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2022-01-11T13:57:58.522 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2022-01-11T13:57:58.522 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:17.0.0-10004.g17f14252.el8.x86_64 21/24 2022-01-11T13:57:58.522 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2022-01-11T13:57:58.523 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:17.0.0-10004.g17f14252.el8.x86_64 23/24 2022-01-11T13:57:58.594 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2022-01-11T13:57:58.594 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-base-2:17.0.0-10004.g17f14252.el8.x86_64 1/24 2022-01-11T13:57:58.595 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-common-2:17.0.0-10004.g17f14252.el8.x86_64 2/24 2022-01-11T13:57:58.595 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-radosgw-2:17.0.0-10004.g17f14252.el8.x86_64 3/24 2022-01-11T13:57:58.595 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-selinux-2:17.0.0-10004.g17f14252.el8.x86_64 4/24 2022-01-11T13:57:58.596 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs2-2:17.0.0-10004.g17f14252.el8.x86_64 5/24 2022-01-11T13:57:58.596 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradosstriper1-2:17.0.0-10004.g17f14252.el8.x86_ 6/24 2022-01-11T13:57:58.596 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librgw2-2:17.0.0-10004.g17f14252.el8.x86_64 7/24 2022-01-11T13:57:58.596 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-argparse-2:17.0.0-10004.g17f14252.el8 8/24 2022-01-11T13:57:58.596 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-common-2:17.0.0-10004.g17f14252.el8.x 9/24 2022-01-11T13:57:58.597 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cephfs-2:17.0.0-10004.g17f14252.el8.x86_64 10/24 2022-01-11T13:57:58.597 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rados-2:17.0.0-10004.g17f14252.el8.x86_64 11/24 2022-01-11T13:57:58.597 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rbd-2:17.0.0-10004.g17f14252.el8.x86_64 12/24 2022-01-11T13:57:58.597 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rgw-2:17.0.0-10004.g17f14252.el8.x86_64 13/24 2022-01-11T13:57:58.598 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/24 2022-01-11T13:57:58.598 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/24 2022-01-11T13:57:58.598 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/24 2022-01-11T13:57:58.598 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/24 2022-01-11T13:57:58.598 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/24 2022-01-11T13:57:58.599 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2022-01-11T13:57:58.599 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2022-01-11T13:57:58.599 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:17.0.0-10004.g17f14252.el8.x86_64 21/24 2022-01-11T13:57:58.599 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2022-01-11T13:57:58.599 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:17.0.0-10004.g17f14252.el8.x86_64 23/24 2022-01-11T13:57:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2022-01-11T13:57:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:57:59.283 INFO:teuthology.orchestra.run.smithi172.stdout:Upgraded: 2022-01-11T13:57:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: librados2-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:57:59.284 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:57:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-01-11T13:57:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: liboath-2.6.2-3.el8.x86_64 2022-01-11T13:57:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-01-11T13:57:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka-0.11.4-1.el8.x86_64 2022-01-11T13:57:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind-1.3.1-3.el8.x86_64 2022-01-11T13:57:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-01-11T13:57:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-01-11T13:57:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:57:59.289 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:57:59.374 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2022-01-11T13:57:59.375 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:57:59.375 INFO:teuthology.orchestra.run.smithi161.stdout:Upgraded: 2022-01-11T13:57:59.375 INFO:teuthology.orchestra.run.smithi161.stdout: librados2-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.375 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.376 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:57:59.376 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:57:59.376 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.376 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.377 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.377 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.377 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-01-11T13:57:59.377 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.378 INFO:teuthology.orchestra.run.smithi161.stdout: liboath-2.6.2-3.el8.x86_64 2022-01-11T13:57:59.378 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-01-11T13:57:59.378 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.378 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka-0.11.4-1.el8.x86_64 2022-01-11T13:57:59.379 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.379 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind-1.3.1-3.el8.x86_64 2022-01-11T13:57:59.379 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-01-11T13:57:59.379 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.380 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.380 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.380 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-01-11T13:57:59.380 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.380 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.381 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:57:59.381 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:57:59.381 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:57:59.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-test 2022-01-11T13:57:59.659 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-test 2022-01-11T13:58:00.009 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:37 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:00.145 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:00:37 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:00.900 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:58:00.901 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:58:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2022-01-11T13:58:00.902 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:58:00.902 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:58:00.902 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test x86_64 2:17.0.0-10004.g17f14252.el8 ceph 45 M 2022-01-11T13:58:00.903 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2022-01-11T13:58:00.903 INFO:teuthology.orchestra.run.smithi172.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2022-01-11T13:58:00.903 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite x86_64 2:17.0.0-10004.g17f14252.el8 ceph 170 k 2022-01-11T13:58:00.903 INFO:teuthology.orchestra.run.smithi172.stdout: libxslt x86_64 1.1.32-6.el8 CentOS-Base 250 k 2022-01-11T13:58:00.903 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-01-11T13:58:00.904 INFO:teuthology.orchestra.run.smithi172.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-01-11T13:58:00.904 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-01-11T13:58:00.904 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:00.904 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:58:00.905 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:58:00.905 INFO:teuthology.orchestra.run.smithi172.stdout:Install 7 Packages 2022-01-11T13:58:00.905 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:00.906 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 46 M 2022-01-11T13:58:00.907 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 214 M 2022-01-11T13:58:00.907 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:58:00.937 INFO:teuthology.orchestra.run.smithi172.stdout:(1/7): jq-1.5-12.el8.x86_64.rpm 4.9 MB/s | 161 kB 00:00 2022-01-11T13:58:00.949 INFO:teuthology.orchestra.run.smithi172.stdout:(2/7): oniguruma-6.8.2-2.el8.x86_64.rpm 15 MB/s | 187 kB 00:00 2022-01-11T13:58:00.954 INFO:teuthology.orchestra.run.smithi172.stdout:(3/7): socat-1.7.4.1-1.el8.x86_64.rpm 70 MB/s | 323 kB 00:00 2022-01-11T13:58:00.957 INFO:teuthology.orchestra.run.smithi172.stdout:(4/7): libxslt-1.1.32-6.el8.x86_64.rpm 68 MB/s | 250 kB 00:00 2022-01-11T13:58:01.021 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:58:01.023 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:58:01.023 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2022-01-11T13:58:01.023 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:58:01.023 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:58:01.023 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test x86_64 2:17.0.0-10004.g17f14252.el8 ceph 45 M 2022-01-11T13:58:01.024 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2022-01-11T13:58:01.024 INFO:teuthology.orchestra.run.smithi161.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2022-01-11T13:58:01.024 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite x86_64 2:17.0.0-10004.g17f14252.el8 ceph 170 k 2022-01-11T13:58:01.024 INFO:teuthology.orchestra.run.smithi161.stdout: libxslt x86_64 1.1.32-6.el8 CentOS-Base 250 k 2022-01-11T13:58:01.025 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-01-11T13:58:01.025 INFO:teuthology.orchestra.run.smithi161.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-01-11T13:58:01.025 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-01-11T13:58:01.025 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:01.025 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:58:01.026 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:58:01.026 INFO:teuthology.orchestra.run.smithi161.stdout:Install 7 Packages 2022-01-11T13:58:01.026 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:01.027 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 46 M 2022-01-11T13:58:01.027 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 214 M 2022-01-11T13:58:01.027 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:58:01.058 INFO:teuthology.orchestra.run.smithi161.stdout:(1/7): jq-1.5-12.el8.x86_64.rpm 4.9 MB/s | 161 kB 00:00 2022-01-11T13:58:01.060 INFO:teuthology.orchestra.run.smithi161.stdout:(2/7): oniguruma-6.8.2-2.el8.x86_64.rpm 65 MB/s | 187 kB 00:00 2022-01-11T13:58:01.074 INFO:teuthology.orchestra.run.smithi161.stdout:(3/7): socat-1.7.4.1-1.el8.x86_64.rpm 71 MB/s | 323 kB 00:00 2022-01-11T13:58:01.082 INFO:teuthology.orchestra.run.smithi161.stdout:(4/7): libxslt-1.1.32-6.el8.x86_64.rpm 71 MB/s | 250 kB 00:00 2022-01-11T13:58:01.084 INFO:teuthology.orchestra.run.smithi172.stdout:(5/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 573 kB/s | 69 kB 00:00 2022-01-11T13:58:01.152 INFO:teuthology.orchestra.run.smithi172.stdout:(6/7): libcephsqlite-17.0.0-10004.g17f14252.el8 688 kB/s | 170 kB 00:00 2022-01-11T13:58:01.192 INFO:teuthology.orchestra.run.smithi161.stdout:(5/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 606 kB/s | 69 kB 00:00 2022-01-11T13:58:01.272 INFO:teuthology.orchestra.run.smithi161.stdout:(6/7): libcephsqlite-17.0.0-10004.g17f14252.el8 688 kB/s | 170 kB 00:00 2022-01-11T13:58:04.658 INFO:teuthology.orchestra.run.smithi172.stdout:(7/7): ceph-test-17.0.0-10004.g17f14252.el8.x86 12 MB/s | 45 MB 00:03 2022-01-11T13:58:04.661 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:58:04.662 INFO:teuthology.orchestra.run.smithi172.stdout:Total 12 MB/s | 46 MB 00:03 2022-01-11T13:58:04.670 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:58:04.694 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:58:04.694 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:58:04.948 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:58:04.951 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:58:05.197 INFO:teuthology.orchestra.run.smithi161.stdout:(7/7): ceph-test-17.0.0-10004.g17f14252.el8.x86 11 MB/s | 45 MB 00:04 2022-01-11T13:58:05.201 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:58:05.202 INFO:teuthology.orchestra.run.smithi161.stdout:Total 11 MB/s | 46 MB 00:04 2022-01-11T13:58:05.210 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:58:05.234 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:58:05.234 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:58:05.467 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:58:05.488 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:58:05.492 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:58:05.658 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libxslt-1.1.32-6.el8.x86_64 1/7 2022-01-11T13:58:05.821 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2022-01-11T13:58:05.980 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:58:05.994 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/7 2022-01-11T13:58:06.126 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libxslt-1.1.32-6.el8.x86_64 1/7 2022-01-11T13:58:06.182 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/7 2022-01-11T13:58:06.267 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/7 2022-01-11T13:58:06.315 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2022-01-11T13:58:06.432 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : jq-1.5-12.el8.x86_64 5/7 2022-01-11T13:58:06.459 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/7 2022-01-11T13:58:06.565 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephsqlite-2:17.0.0-10004.g17f14252.el8.x86_64 6/7 2022-01-11T13:58:06.652 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/7 2022-01-11T13:58:06.734 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/7 2022-01-11T13:58:06.875 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : jq-1.5-12.el8.x86_64 5/7 2022-01-11T13:58:07.001 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephsqlite-2:17.0.0-10004.g17f14252.el8.x86_64 6/7 2022-01-11T13:58:11.013 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephsqlite-2:17.0.0-10004.g17f14252.el8.x86_64 6/7 2022-01-11T13:58:11.272 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-test-2:17.0.0-10004.g17f14252.el8.x86_64 7/7 2022-01-11T13:58:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephsqlite-2:17.0.0-10004.g17f14252.el8.x86_64 6/7 2022-01-11T13:58:11.515 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-test-2:17.0.0-10004.g17f14252.el8.x86_64 7/7 2022-01-11T13:58:11.515 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-test-2:17.0.0-10004.g17f14252.el8.x86_64 1/7 2022-01-11T13:58:11.516 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephsqlite-2:17.0.0-10004.g17f14252.el8.x86_64 2/7 2022-01-11T13:58:11.516 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : jq-1.5-12.el8.x86_64 3/7 2022-01-11T13:58:11.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/7 2022-01-11T13:58:11.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/7 2022-01-11T13:58:11.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libxslt-1.1.32-6.el8.x86_64 6/7 2022-01-11T13:58:11.716 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-test-2:17.0.0-10004.g17f14252.el8.x86_64 7/7 2022-01-11T13:58:11.904 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 7/7 2022-01-11T13:58:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:11.905 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:58:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test-2:17.0.0-10004.g17f14252.el8.x86_64 jq-1.5-12.el8.x86_64 2022-01-11T13:58:11.905 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite-2:17.0.0-10004.g17f14252.el8.x86_64 libxslt-1.1.32-6.el8.x86_64 2022-01-11T13:58:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2022-01-11T13:58:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-01-11T13:58:11.906 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:11.906 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:58:11.953 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-test-2:17.0.0-10004.g17f14252.el8.x86_64 7/7 2022-01-11T13:58:11.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-test-2:17.0.0-10004.g17f14252.el8.x86_64 1/7 2022-01-11T13:58:11.953 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephsqlite-2:17.0.0-10004.g17f14252.el8.x86_64 2/7 2022-01-11T13:58:11.954 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : jq-1.5-12.el8.x86_64 3/7 2022-01-11T13:58:11.954 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/7 2022-01-11T13:58:11.954 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/7 2022-01-11T13:58:11.954 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libxslt-1.1.32-6.el8.x86_64 6/7 2022-01-11T13:58:12.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph 2022-01-11T13:58:12.371 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 7/7 2022-01-11T13:58:12.371 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:12.372 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:58:12.372 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test-2:17.0.0-10004.g17f14252.el8.x86_64 jq-1.5-12.el8.x86_64 2022-01-11T13:58:12.372 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite-2:17.0.0-10004.g17f14252.el8.x86_64 libxslt-1.1.32-6.el8.x86_64 2022-01-11T13:58:12.372 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2022-01-11T13:58:12.373 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-01-11T13:58:12.373 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:12.373 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:58:12.555 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph 2022-01-11T13:58:12.605 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:49 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:13.033 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:00:50 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:13.928 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:58:13.932 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2022-01-11T13:58:13.932 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2022-01-11T13:58:13.933 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2022-01-11T13:58:13.933 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:58:13.933 INFO:teuthology.orchestra.run.smithi172.stdout: ceph x86_64 2:17.0.0-10004.g17f14252.el8 ceph 6.5 k 2022-01-11T13:58:13.934 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2022-01-11T13:58:13.934 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds x86_64 2:17.0.0-10004.g17f14252.el8 ceph 2.3 M 2022-01-11T13:58:13.935 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr x86_64 2:17.0.0-10004.g17f14252.el8 ceph 1.5 M 2022-01-11T13:58:13.935 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 218 k 2022-01-11T13:58:13.936 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon x86_64 2:17.0.0-10004.g17f14252.el8 ceph 3.9 M 2022-01-11T13:58:13.936 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd x86_64 2:17.0.0-10004.g17f14252.el8 ceph 17 M 2022-01-11T13:58:13.936 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-01-11T13:58:13.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-01-11T13:58:13.937 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2022-01-11T13:58:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2022-01-11T13:58:13.938 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-01-11T13:58:13.939 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-01-11T13:58:13.939 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2022-01-11T13:58:13.940 INFO:teuthology.orchestra.run.smithi172.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2022-01-11T13:58:13.940 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-01-11T13:58:13.940 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-01-11T13:58:13.941 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-01-11T13:58:13.941 INFO:teuthology.orchestra.run.smithi172.stdout: python3-lxml x86_64 4.2.3-3.el8 CentOS-AppStream 1.5 M 2022-01-11T13:58:13.942 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-01-11T13:58:13.942 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-01-11T13:58:13.942 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-01-11T13:58:13.943 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-01-11T13:58:13.943 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-01-11T13:58:13.943 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-01-11T13:58:13.943 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2022-01-11T13:58:13.944 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2022-01-11T13:58:13.944 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-01-11T13:58:13.944 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2022-01-11T13:58:13.945 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-01-11T13:58:13.945 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-01-11T13:58:13.945 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-01-11T13:58:13.946 INFO:teuthology.orchestra.run.smithi172.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-01-11T13:58:13.946 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2022-01-11T13:58:13.947 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-01-11T13:58:13.947 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2022-01-11T13:58:13.947 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-01-11T13:58:13.947 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-01-11T13:58:13.948 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-01-11T13:58:13.948 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-01-11T13:58:13.948 INFO:teuthology.orchestra.run.smithi172.stdout:Installing weak dependencies: 2022-01-11T13:58:13.948 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2022-01-11T13:58:13.948 INFO:teuthology.orchestra.run.smithi172.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2022-01-11T13:58:13.949 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:13.949 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:58:13.949 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2022-01-11T13:58:13.949 INFO:teuthology.orchestra.run.smithi172.stdout:Install 41 Packages 2022-01-11T13:58:13.950 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:13.951 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 30 M 2022-01-11T13:58:13.951 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 120 M 2022-01-11T13:58:13.952 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:58:14.088 INFO:teuthology.orchestra.run.smithi172.stdout:(1/41): ceph-17.0.0-10004.g17f14252.el8.x86_64. 43 kB/s | 6.5 kB 00:00 2022-01-11T13:58:14.251 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:58:14.254 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2022-01-11T13:58:14.255 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2022-01-11T13:58:14.255 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2022-01-11T13:58:14.255 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:58:14.256 INFO:teuthology.orchestra.run.smithi161.stdout: ceph x86_64 2:17.0.0-10004.g17f14252.el8 ceph 6.5 k 2022-01-11T13:58:14.256 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2022-01-11T13:58:14.256 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds x86_64 2:17.0.0-10004.g17f14252.el8 ceph 2.3 M 2022-01-11T13:58:14.257 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr x86_64 2:17.0.0-10004.g17f14252.el8 ceph 1.5 M 2022-01-11T13:58:14.257 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 218 k 2022-01-11T13:58:14.257 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon x86_64 2:17.0.0-10004.g17f14252.el8 ceph 3.9 M 2022-01-11T13:58:14.257 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd x86_64 2:17.0.0-10004.g17f14252.el8 ceph 17 M 2022-01-11T13:58:14.258 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-01-11T13:58:14.258 INFO:teuthology.orchestra.run.smithi161.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-01-11T13:58:14.258 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2022-01-11T13:58:14.258 INFO:teuthology.orchestra.run.smithi161.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2022-01-11T13:58:14.259 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-01-11T13:58:14.259 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-01-11T13:58:14.259 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2022-01-11T13:58:14.259 INFO:teuthology.orchestra.run.smithi161.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2022-01-11T13:58:14.259 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-01-11T13:58:14.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-01-11T13:58:14.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-01-11T13:58:14.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-lxml x86_64 4.2.3-3.el8 CentOS-AppStream 1.5 M 2022-01-11T13:58:14.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-01-11T13:58:14.261 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-01-11T13:58:14.261 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-01-11T13:58:14.261 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-01-11T13:58:14.261 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-01-11T13:58:14.261 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-01-11T13:58:14.262 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2022-01-11T13:58:14.262 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2022-01-11T13:58:14.262 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-01-11T13:58:14.263 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2022-01-11T13:58:14.263 INFO:teuthology.orchestra.run.smithi161.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-01-11T13:58:14.263 INFO:teuthology.orchestra.run.smithi161.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-01-11T13:58:14.263 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-01-11T13:58:14.263 INFO:teuthology.orchestra.run.smithi161.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-01-11T13:58:14.264 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2022-01-11T13:58:14.264 INFO:teuthology.orchestra.run.smithi161.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-01-11T13:58:14.264 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2022-01-11T13:58:14.264 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-01-11T13:58:14.265 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-01-11T13:58:14.265 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-01-11T13:58:14.265 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-01-11T13:58:14.265 INFO:teuthology.orchestra.run.smithi161.stdout:Installing weak dependencies: 2022-01-11T13:58:14.265 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2022-01-11T13:58:14.266 INFO:teuthology.orchestra.run.smithi161.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2022-01-11T13:58:14.266 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:14.266 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:58:14.266 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2022-01-11T13:58:14.267 INFO:teuthology.orchestra.run.smithi161.stdout:Install 41 Packages 2022-01-11T13:58:14.267 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:14.268 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 30 M 2022-01-11T13:58:14.268 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 120 M 2022-01-11T13:58:14.268 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:58:14.372 INFO:teuthology.orchestra.run.smithi172.stdout:(2/41): ceph-mgr-17.0.0-10004.g17f14252.el8.x86 3.5 MB/s | 1.5 MB 00:00 2022-01-11T13:58:14.417 INFO:teuthology.orchestra.run.smithi172.stdout:(3/41): ceph-mds-17.0.0-10004.g17f14252.el8.x86 4.7 MB/s | 2.3 MB 00:00 2022-01-11T13:58:14.484 INFO:teuthology.orchestra.run.smithi172.stdout:(4/41): ceph-mgr-modules-core-17.0.0-10004.g17f 3.3 MB/s | 218 kB 00:00 2022-01-11T13:58:14.514 INFO:teuthology.orchestra.run.smithi172.stdout:(5/41): python3-html5lib-0.999999999-6.el8.noar 6.7 MB/s | 214 kB 00:00 2022-01-11T13:58:14.532 INFO:teuthology.orchestra.run.smithi172.stdout:(6/41): python3-lxml-4.2.3-3.el8.x86_64.rpm 81 MB/s | 1.5 MB 00:00 2022-01-11T13:58:14.535 INFO:teuthology.orchestra.run.smithi172.stdout:(7/41): python3-mako-1.0.6-13.el8.noarch.rpm 57 MB/s | 157 kB 00:00 2022-01-11T13:58:14.537 INFO:teuthology.orchestra.run.smithi172.stdout:(8/41): python3-markupsafe-0.23-19.el8.x86_64.r 30 MB/s | 39 kB 00:00 2022-01-11T13:58:14.537 INFO:teuthology.orchestra.run.smithi161.stdout:(1/41): ceph-17.0.0-10004.g17f14252.el8.x86_64. 23 kB/s | 6.5 kB 00:00 2022-01-11T13:58:14.539 INFO:teuthology.orchestra.run.smithi172.stdout:(9/41): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 53 MB/s | 103 kB 00:00 2022-01-11T13:58:14.540 INFO:teuthology.orchestra.run.smithi172.stdout:(10/41): python3-pytz-2017.2-9.el8.noarch.rpm 34 MB/s | 54 kB 00:00 2022-01-11T13:58:14.542 INFO:teuthology.orchestra.run.smithi172.stdout:(11/41): python3-webencodings-0.5.1-6.el8.noarc 24 MB/s | 27 kB 00:00 2022-01-11T13:58:14.556 INFO:teuthology.orchestra.run.smithi172.stdout:(12/41): ceph-mon-17.0.0-10004.g17f14252.el8.x8 8.2 MB/s | 3.9 MB 00:00 2022-01-11T13:58:14.558 INFO:teuthology.orchestra.run.smithi172.stdout:(13/41): python3-werkzeug-0.12.2-4.el8.noarch.r 28 MB/s | 457 kB 00:00 2022-01-11T13:58:14.566 INFO:teuthology.orchestra.run.smithi172.stdout:(14/41): python3-chardet-3.0.4-7.el8.noarch.rpm 58 MB/s | 195 kB 00:00 2022-01-11T13:58:14.573 INFO:teuthology.orchestra.run.smithi172.stdout:(15/41): python3-cryptography-3.2.1-5.el8.x86_6 78 MB/s | 559 kB 00:00 2022-01-11T13:58:14.575 INFO:teuthology.orchestra.run.smithi172.stdout:(16/41): python3-idna-2.5-5.el8.noarch.rpm 50 MB/s | 97 kB 00:00 2022-01-11T13:58:14.578 INFO:teuthology.orchestra.run.smithi172.stdout:(17/41): python3-pycparser-2.14-14.el8.noarch.r 55 MB/s | 109 kB 00:00 2022-01-11T13:58:14.579 INFO:teuthology.orchestra.run.smithi172.stdout:(18/41): python3-pysocks-1.6.8-3.el8.noarch.rpm 30 MB/s | 34 kB 00:00 2022-01-11T13:58:14.581 INFO:teuthology.orchestra.run.smithi172.stdout:(19/41): python3-requests-2.20.0-2.1.el8_1.noar 56 MB/s | 123 kB 00:00 2022-01-11T13:58:14.584 INFO:teuthology.orchestra.run.smithi172.stdout:(20/41): python3-urllib3-1.24.2-5.el8.noarch.rp 55 MB/s | 177 kB 00:00 2022-01-11T13:58:14.588 INFO:teuthology.orchestra.run.smithi172.stdout:(21/41): python3-cffi-1.11.5-5.el8.x86_64.rpm 7.2 MB/s | 237 kB 00:00 2022-01-11T13:58:14.699 INFO:teuthology.orchestra.run.smithi172.stdout:(22/41): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 383 kB/s | 44 kB 00:00 2022-01-11T13:58:14.730 INFO:teuthology.orchestra.run.smithi172.stdout:(23/41): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.3 MB/s | 185 kB 00:00 2022-01-11T13:58:14.821 INFO:teuthology.orchestra.run.smithi172.stdout:(24/41): python3-cherrypy-18.4.0-1.el8.noarch.r 4.1 MB/s | 384 kB 00:00 2022-01-11T13:58:14.833 INFO:teuthology.orchestra.run.smithi172.stdout:(25/41): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2022-01-11T13:58:14.849 INFO:teuthology.orchestra.run.smithi172.stdout:(26/41): python3-cssselect-0.9.2-10.el8.noarch. 1.4 MB/s | 40 kB 00:00 2022-01-11T13:58:15.058 INFO:teuthology.orchestra.run.smithi161.stdout:(2/41): ceph-mgr-17.0.0-10004.g17f14252.el8.x86 1.9 MB/s | 1.5 MB 00:00 2022-01-11T13:58:15.148 INFO:teuthology.orchestra.run.smithi161.stdout:(3/41): ceph-mds-17.0.0-10004.g17f14252.el8.x86 2.6 MB/s | 2.3 MB 00:00 2022-01-11T13:58:15.226 INFO:teuthology.orchestra.run.smithi161.stdout:(4/41): ceph-mgr-modules-core-17.0.0-10004.g17f 2.8 MB/s | 218 kB 00:00 2022-01-11T13:58:15.258 INFO:teuthology.orchestra.run.smithi161.stdout:(5/41): python3-html5lib-0.999999999-6.el8.noar 6.4 MB/s | 214 kB 00:00 2022-01-11T13:58:15.276 INFO:teuthology.orchestra.run.smithi161.stdout:(6/41): python3-lxml-4.2.3-3.el8.x86_64.rpm 83 MB/s | 1.5 MB 00:00 2022-01-11T13:58:15.279 INFO:teuthology.orchestra.run.smithi161.stdout:(7/41): python3-mako-1.0.6-13.el8.noarch.rpm 58 MB/s | 157 kB 00:00 2022-01-11T13:58:15.280 INFO:teuthology.orchestra.run.smithi161.stdout:(8/41): python3-markupsafe-0.23-19.el8.x86_64.r 29 MB/s | 39 kB 00:00 2022-01-11T13:58:15.283 INFO:teuthology.orchestra.run.smithi161.stdout:(9/41): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 48 MB/s | 103 kB 00:00 2022-01-11T13:58:15.284 INFO:teuthology.orchestra.run.smithi161.stdout:(10/41): python3-pytz-2017.2-9.el8.noarch.rpm 37 MB/s | 54 kB 00:00 2022-01-11T13:58:15.285 INFO:teuthology.orchestra.run.smithi161.stdout:(11/41): python3-webencodings-0.5.1-6.el8.noarc 24 MB/s | 27 kB 00:00 2022-01-11T13:58:15.292 INFO:teuthology.orchestra.run.smithi161.stdout:(12/41): python3-werkzeug-0.12.2-4.el8.noarch.r 73 MB/s | 457 kB 00:00 2022-01-11T13:58:15.295 INFO:teuthology.orchestra.run.smithi161.stdout:(13/41): python3-cffi-1.11.5-5.el8.x86_64.rpm 67 MB/s | 237 kB 00:00 2022-01-11T13:58:15.299 INFO:teuthology.orchestra.run.smithi161.stdout:(14/41): python3-chardet-3.0.4-7.el8.noarch.rpm 62 MB/s | 195 kB 00:00 2022-01-11T13:58:15.306 INFO:teuthology.orchestra.run.smithi161.stdout:(15/41): python3-cryptography-3.2.1-5.el8.x86_6 74 MB/s | 559 kB 00:00 2022-01-11T13:58:15.308 INFO:teuthology.orchestra.run.smithi161.stdout:(16/41): python3-idna-2.5-5.el8.noarch.rpm 53 MB/s | 97 kB 00:00 2022-01-11T13:58:15.310 INFO:teuthology.orchestra.run.smithi161.stdout:(17/41): python3-pycparser-2.14-14.el8.noarch.r 53 MB/s | 109 kB 00:00 2022-01-11T13:58:15.312 INFO:teuthology.orchestra.run.smithi161.stdout:(18/41): python3-pysocks-1.6.8-3.el8.noarch.rpm 24 MB/s | 34 kB 00:00 2022-01-11T13:58:15.314 INFO:teuthology.orchestra.run.smithi161.stdout:(19/41): python3-requests-2.20.0-2.1.el8_1.noar 54 MB/s | 123 kB 00:00 2022-01-11T13:58:15.318 INFO:teuthology.orchestra.run.smithi161.stdout:(20/41): python3-urllib3-1.24.2-5.el8.noarch.rp 63 MB/s | 177 kB 00:00 2022-01-11T13:58:15.400 INFO:teuthology.orchestra.run.smithi172.stdout:(27/41): ceph-osd-17.0.0-10004.g17f14252.el8.x8 16 MB/s | 17 MB 00:01 2022-01-11T13:58:15.433 INFO:teuthology.orchestra.run.smithi161.stdout:(21/41): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 376 kB/s | 44 kB 00:00 2022-01-11T13:58:15.519 INFO:teuthology.orchestra.run.smithi161.stdout:(22/41): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.1 MB/s | 185 kB 00:00 2022-01-11T13:58:15.541 INFO:teuthology.orchestra.run.smithi161.stdout:(23/41): ceph-mon-17.0.0-10004.g17f14252.el8.x8 3.8 MB/s | 3.9 MB 00:01 2022-01-11T13:58:15.550 INFO:teuthology.orchestra.run.smithi161.stdout:(24/41): python3-cheroot-8.5.2-1.el8.noarch.rpm 5.5 MB/s | 173 kB 00:00 2022-01-11T13:58:15.579 INFO:teuthology.orchestra.run.smithi161.stdout:(25/41): python3-cssselect-0.9.2-10.el8.noarch. 1.4 MB/s | 40 kB 00:00 2022-01-11T13:58:15.587 INFO:teuthology.orchestra.run.smithi172.stdout:(28/41): python3-jaraco-6.2-6.el8.noarch.rpm 15 kB/s | 11 kB 00:00 2022-01-11T13:58:15.608 INFO:teuthology.orchestra.run.smithi161.stdout:(26/41): python3-jaraco-6.2-6.el8.noarch.rpm 389 kB/s | 11 kB 00:00 2022-01-11T13:58:15.638 INFO:teuthology.orchestra.run.smithi172.stdout:(29/41): python3-jaraco-functools-2.0-4.el8.noa 22 kB/s | 18 kB 00:00 2022-01-11T13:58:15.639 INFO:teuthology.orchestra.run.smithi161.stdout:(27/41): python3-jaraco-functools-2.0-4.el8.noa 571 kB/s | 18 kB 00:00 2022-01-11T13:58:15.663 INFO:teuthology.orchestra.run.smithi172.stdout:(30/41): python3-more-itertools-7.2.0-3.el8.noa 772 kB/s | 59 kB 00:00 2022-01-11T13:58:15.668 INFO:teuthology.orchestra.run.smithi161.stdout:(28/41): python3-logutils-0.3.5-11.el8.noarch.r 1.6 MB/s | 49 kB 00:00 2022-01-11T13:58:15.688 INFO:teuthology.orchestra.run.smithi172.stdout:(31/41): python3-logutils-0.3.5-11.el8.noarch.r 169 kB/s | 49 kB 00:00 2022-01-11T13:58:15.691 INFO:teuthology.orchestra.run.smithi172.stdout:(32/41): python3-portend-2.6-1.el8.noarch.rpm 559 kB/s | 16 kB 00:00 2022-01-11T13:58:15.697 INFO:teuthology.orchestra.run.smithi161.stdout:(29/41): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2022-01-11T13:58:15.714 INFO:teuthology.orchestra.run.smithi161.stdout:(30/41): python3-cherrypy-18.4.0-1.el8.noarch.r 2.2 MB/s | 384 kB 00:00 2022-01-11T13:58:15.715 INFO:teuthology.orchestra.run.smithi172.stdout:(33/41): python3-simplegeneric-0.8.1-17.el8.noa 676 kB/s | 19 kB 00:00 2022-01-11T13:58:15.720 INFO:teuthology.orchestra.run.smithi172.stdout:(34/41): python3-singledispatch-3.4.0.3-18.el8. 851 kB/s | 24 kB 00:00 2022-01-11T13:58:15.729 INFO:teuthology.orchestra.run.smithi161.stdout:(31/41): python3-pecan-1.3.2-9.el8.noarch.rpm 8.6 MB/s | 283 kB 00:00 2022-01-11T13:58:15.745 INFO:teuthology.orchestra.run.smithi172.stdout:(35/41): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2022-01-11T13:58:15.745 INFO:teuthology.orchestra.run.smithi161.stdout:(32/41): python3-portend-2.6-1.el8.noarch.rpm 523 kB/s | 16 kB 00:00 2022-01-11T13:58:15.747 INFO:teuthology.orchestra.run.smithi172.stdout:(36/41): python3-trustme-0.6.0-4.el8.noarch.rpm 994 kB/s | 27 kB 00:00 2022-01-11T13:58:15.758 INFO:teuthology.orchestra.run.smithi161.stdout:(33/41): python3-simplegeneric-0.8.1-17.el8.noa 660 kB/s | 19 kB 00:00 2022-01-11T13:58:15.766 INFO:teuthology.orchestra.run.smithi172.stdout:(37/41): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2022-01-11T13:58:15.773 INFO:teuthology.orchestra.run.smithi161.stdout:(34/41): python3-singledispatch-3.4.0.3-18.el8. 844 kB/s | 24 kB 00:00 2022-01-11T13:58:15.788 INFO:teuthology.orchestra.run.smithi161.stdout:(35/41): python3-tempora-1.14.1-5.el8.noarch.rp 989 kB/s | 29 kB 00:00 2022-01-11T13:58:15.795 INFO:teuthology.orchestra.run.smithi172.stdout:(38/41): python3-webtest-2.0.33-1.el8.noarch.rp 2.9 MB/s | 85 kB 00:00 2022-01-11T13:58:15.802 INFO:teuthology.orchestra.run.smithi161.stdout:(36/41): python3-trustme-0.6.0-4.el8.noarch.rpm 936 kB/s | 27 kB 00:00 2022-01-11T13:58:15.811 INFO:teuthology.orchestra.run.smithi172.stdout:(39/41): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.9 MB/s | 251 kB 00:00 2022-01-11T13:58:15.824 INFO:teuthology.orchestra.run.smithi172.stdout:(40/41): python3-zc-lockfile-2.0-2.el8.noarch.r 789 kB/s | 23 kB 00:00 2022-01-11T13:58:15.833 INFO:teuthology.orchestra.run.smithi172.stdout:(41/41): python3-waitress-1.2.1-2.el8.1.noarch. 2.7 MB/s | 241 kB 00:00 2022-01-11T13:58:15.834 INFO:teuthology.orchestra.run.smithi161.stdout:(37/41): python3-waitress-1.2.1-2.el8.1.noarch. 5.3 MB/s | 241 kB 00:00 2022-01-11T13:58:15.835 INFO:teuthology.orchestra.run.smithi161.stdout:(38/41): python3-webob-1.8.5-1.el8.1.noarch.rpm 7.4 MB/s | 251 kB 00:00 2022-01-11T13:58:15.836 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:58:15.837 INFO:teuthology.orchestra.run.smithi172.stdout:Total 16 MB/s | 30 MB 00:01 2022-01-11T13:58:15.863 INFO:teuthology.orchestra.run.smithi161.stdout:(39/41): python3-webtest-2.0.33-1.el8.noarch.rp 2.8 MB/s | 85 kB 00:00 2022-01-11T13:58:15.863 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:58:15.865 INFO:teuthology.orchestra.run.smithi161.stdout:(40/41): python3-zc-lockfile-2.0-2.el8.noarch.r 778 kB/s | 23 kB 00:00 2022-01-11T13:58:15.903 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:58:15.903 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:58:16.219 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:58:16.222 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:58:16.413 INFO:teuthology.orchestra.run.smithi161.stdout:(41/41): ceph-osd-17.0.0-10004.g17f14252.el8.x8 12 MB/s | 17 MB 00:01 2022-01-11T13:58:16.416 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:58:16.417 INFO:teuthology.orchestra.run.smithi161.stdout:Total 14 MB/s | 30 MB 00:02 2022-01-11T13:58:16.442 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:58:16.481 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:58:16.481 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:58:16.779 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:58:16.782 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:58:17.722 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:58:17.898 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/41 2022-01-11T13:58:18.117 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/41 2022-01-11T13:58:18.226 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:58:18.264 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-idna-2.5-5.el8.noarch 3/41 2022-01-11T13:58:18.459 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/41 2022-01-11T13:58:18.489 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/41 2022-01-11T13:58:18.645 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/41 2022-01-11T13:58:18.695 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/41 2022-01-11T13:58:18.826 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/41 2022-01-11T13:58:18.833 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-idna-2.5-5.el8.noarch 3/41 2022-01-11T13:58:19.008 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/41 2022-01-11T13:58:19.051 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/41 2022-01-11T13:58:19.216 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/41 2022-01-11T13:58:19.240 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/41 2022-01-11T13:58:19.354 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/41 2022-01-11T13:58:19.361 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/41 2022-01-11T13:58:19.494 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/41 2022-01-11T13:58:19.535 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/41 2022-01-11T13:58:19.680 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 11/41 2022-01-11T13:58:19.703 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/41 2022-01-11T13:58:19.847 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/41 2022-01-11T13:58:19.915 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 12/41 2022-01-11T13:58:19.995 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/41 2022-01-11T13:58:20.165 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 13/41 2022-01-11T13:58:20.182 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 11/41 2022-01-11T13:58:20.397 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 12/41 2022-01-11T13:58:20.413 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 14/41 2022-01-11T13:58:20.640 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 13/41 2022-01-11T13:58:20.776 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 15/41 2022-01-11T13:58:20.855 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 14/41 2022-01-11T13:58:21.051 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 16/41 2022-01-11T13:58:21.243 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 15/41 2022-01-11T13:58:21.255 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 17/41 2022-01-11T13:58:21.450 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 18/41 2022-01-11T13:58:21.520 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 16/41 2022-01-11T13:58:21.672 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 19/41 2022-01-11T13:58:21.691 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 17/41 2022-01-11T13:58:21.927 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 18/41 2022-01-11T13:58:21.940 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 20/41 2022-01-11T13:58:22.134 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 19/41 2022-01-11T13:58:22.161 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 21/41 2022-01-11T13:58:22.433 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 20/41 2022-01-11T13:58:22.434 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 22/41 2022-01-11T13:58:22.638 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 21/41 2022-01-11T13:58:22.716 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/41 2022-01-11T13:58:22.912 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 22/41 2022-01-11T13:58:22.926 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 24/41 2022-01-11T13:58:23.119 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/41 2022-01-11T13:58:23.143 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 25/41 2022-01-11T13:58:23.363 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 24/41 2022-01-11T13:58:23.536 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 26/41 2022-01-11T13:58:23.563 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 25/41 2022-01-11T13:58:23.802 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-lxml-4.2.3-3.el8.x86_64 27/41 2022-01-11T13:58:23.948 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 26/41 2022-01-11T13:58:24.002 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 28/41 2022-01-11T13:58:24.205 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-lxml-4.2.3-3.el8.x86_64 27/41 2022-01-11T13:58:24.224 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 29/41 2022-01-11T13:58:24.381 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 28/41 2022-01-11T13:58:24.426 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 30/41 2022-01-11T13:58:24.594 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 29/41 2022-01-11T13:58:24.727 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-portend-2.6-1.el8.noarch 31/41 2022-01-11T13:58:24.771 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 30/41 2022-01-11T13:58:24.951 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 32/41 2022-01-11T13:58:25.026 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-portend-2.6-1.el8.noarch 31/41 2022-01-11T13:58:25.177 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 33/41 2022-01-11T13:58:25.221 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 32/41 2022-01-11T13:58:25.454 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 33/41 2022-01-11T13:58:25.508 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 34/41 2022-01-11T13:58:25.771 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 34/41 2022-01-11T13:58:25.855 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 35/41 2022-01-11T13:58:26.099 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 35/41 2022-01-11T13:58:26.221 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-10004.g17f14252.el8 36/41 2022-01-11T13:58:26.417 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-2:17.0.0-10004.g17f14252.el8.x86_64 37/41 2022-01-11T13:58:26.451 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-10004.g17f14252.el8 36/41 2022-01-11T13:58:26.629 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-2:17.0.0-10004.g17f14252.el8.x86_64 37/41 2022-01-11T13:58:27.911 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-2:17.0.0-10004.g17f14252.el8.x86_64 37/41 2022-01-11T13:58:28.092 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-osd-2:17.0.0-10004.g17f14252.el8.x86_64 38/41 2022-01-11T13:58:28.129 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-2:17.0.0-10004.g17f14252.el8.x86_64 37/41 2022-01-11T13:58:28.312 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-osd-2:17.0.0-10004.g17f14252.el8.x86_64 38/41 2022-01-11T13:58:28.486 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-osd-2:17.0.0-10004.g17f14252.el8.x86_64 38/41 2022-01-11T13:58:28.704 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mon-2:17.0.0-10004.g17f14252.el8.x86_64 39/41 2022-01-11T13:58:28.707 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-osd-2:17.0.0-10004.g17f14252.el8.x86_64 38/41 2022-01-11T13:58:28.921 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mon-2:17.0.0-10004.g17f14252.el8.x86_64 39/41 2022-01-11T13:58:28.924 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mon-2:17.0.0-10004.g17f14252.el8.x86_64 39/41 2022-01-11T13:58:29.095 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mds-2:17.0.0-10004.g17f14252.el8.x86_64 40/41 2022-01-11T13:58:29.112 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mds-2:17.0.0-10004.g17f14252.el8.x86_64 40/41 2022-01-11T13:58:29.141 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mon-2:17.0.0-10004.g17f14252.el8.x86_64 39/41 2022-01-11T13:58:29.315 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mds-2:17.0.0-10004.g17f14252.el8.x86_64 40/41 2022-01-11T13:58:29.331 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mds-2:17.0.0-10004.g17f14252.el8.x86_64 40/41 2022-01-11T13:58:29.386 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-2:17.0.0-10004.g17f14252.el8.x86_64 41/41 2022-01-11T13:58:29.506 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-2:17.0.0-10004.g17f14252.el8.x86_64 41/41 2022-01-11T13:58:29.877 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-2:17.0.0-10004.g17f14252.el8.x86_64 41/41 2022-01-11T13:58:29.877 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-2:17.0.0-10004.g17f14252.el8.x86_64 1/41 2022-01-11T13:58:29.877 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mds-2:17.0.0-10004.g17f14252.el8.x86_64 2/41 2022-01-11T13:58:29.878 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-2:17.0.0-10004.g17f14252.el8.x86_64 3/41 2022-01-11T13:58:29.878 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mon-2:17.0.0-10004.g17f14252.el8.x86_64 4/41 2022-01-11T13:58:29.878 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-osd-2:17.0.0-10004.g17f14252.el8.x86_64 5/41 2022-01-11T13:58:29.878 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-10004.g17f14252.el8 6/41 2022-01-11T13:58:29.879 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/41 2022-01-11T13:58:29.879 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-lxml-4.2.3-3.el8.x86_64 8/41 2022-01-11T13:58:29.879 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/41 2022-01-11T13:58:29.879 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/41 2022-01-11T13:58:29.880 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 11/41 2022-01-11T13:58:29.880 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 12/41 2022-01-11T13:58:29.880 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 13/41 2022-01-11T13:58:29.880 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 14/41 2022-01-11T13:58:29.880 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 15/41 2022-01-11T13:58:29.881 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 16/41 2022-01-11T13:58:29.881 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 17/41 2022-01-11T13:58:29.881 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-idna-2.5-5.el8.noarch 18/41 2022-01-11T13:58:29.881 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 19/41 2022-01-11T13:58:29.882 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 20/41 2022-01-11T13:58:29.882 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 21/41 2022-01-11T13:58:29.882 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 22/41 2022-01-11T13:58:29.882 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 23/41 2022-01-11T13:58:29.883 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/41 2022-01-11T13:58:29.883 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 25/41 2022-01-11T13:58:29.883 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 26/41 2022-01-11T13:58:29.883 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 27/41 2022-01-11T13:58:29.884 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 28/41 2022-01-11T13:58:29.884 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 29/41 2022-01-11T13:58:29.884 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 30/41 2022-01-11T13:58:29.884 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 31/41 2022-01-11T13:58:29.885 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 32/41 2022-01-11T13:58:29.885 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-portend-2.6-1.el8.noarch 33/41 2022-01-11T13:58:29.885 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 34/41 2022-01-11T13:58:29.885 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 35/41 2022-01-11T13:58:29.886 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 36/41 2022-01-11T13:58:29.886 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 37/41 2022-01-11T13:58:29.886 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 38/41 2022-01-11T13:58:29.886 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 39/41 2022-01-11T13:58:29.887 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 40/41 2022-01-11T13:58:30.120 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-2:17.0.0-10004.g17f14252.el8.x86_64 41/41 2022-01-11T13:58:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-2:17.0.0-10004.g17f14252.el8.x86_64 1/41 2022-01-11T13:58:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mds-2:17.0.0-10004.g17f14252.el8.x86_64 2/41 2022-01-11T13:58:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-2:17.0.0-10004.g17f14252.el8.x86_64 3/41 2022-01-11T13:58:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mon-2:17.0.0-10004.g17f14252.el8.x86_64 4/41 2022-01-11T13:58:30.121 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-osd-2:17.0.0-10004.g17f14252.el8.x86_64 5/41 2022-01-11T13:58:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-10004.g17f14252.el8 6/41 2022-01-11T13:58:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/41 2022-01-11T13:58:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-lxml-4.2.3-3.el8.x86_64 8/41 2022-01-11T13:58:30.122 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/41 2022-01-11T13:58:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/41 2022-01-11T13:58:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 11/41 2022-01-11T13:58:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 12/41 2022-01-11T13:58:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 13/41 2022-01-11T13:58:30.123 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 14/41 2022-01-11T13:58:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 15/41 2022-01-11T13:58:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 16/41 2022-01-11T13:58:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 17/41 2022-01-11T13:58:30.124 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-idna-2.5-5.el8.noarch 18/41 2022-01-11T13:58:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 19/41 2022-01-11T13:58:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 20/41 2022-01-11T13:58:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 21/41 2022-01-11T13:58:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 22/41 2022-01-11T13:58:30.125 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 23/41 2022-01-11T13:58:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/41 2022-01-11T13:58:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 25/41 2022-01-11T13:58:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 26/41 2022-01-11T13:58:30.126 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 27/41 2022-01-11T13:58:30.127 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 28/41 2022-01-11T13:58:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 29/41 2022-01-11T13:58:30.128 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 30/41 2022-01-11T13:58:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 31/41 2022-01-11T13:58:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 32/41 2022-01-11T13:58:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-portend-2.6-1.el8.noarch 33/41 2022-01-11T13:58:30.129 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 34/41 2022-01-11T13:58:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 35/41 2022-01-11T13:58:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 36/41 2022-01-11T13:58:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 37/41 2022-01-11T13:58:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 38/41 2022-01-11T13:58:30.130 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 39/41 2022-01-11T13:58:30.131 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 40/41 2022-01-11T13:58:31.041 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 41/41 2022-01-11T13:58:31.041 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:31.042 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:58:31.042 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:58:31.042 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:58:31.042 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:58:31.043 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:58:31.043 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:58:31.043 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:58:31.043 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-01-11T13:58:31.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-01-11T13:58:31.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-01-11T13:58:31.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-chardet-3.0.4-7.el8.noarch 2022-01-11T13:58:31.044 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-01-11T13:58:31.045 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-01-11T13:58:31.045 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-01-11T13:58:31.046 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cssselect-0.9.2-10.el8.noarch 2022-01-11T13:58:31.046 INFO:teuthology.orchestra.run.smithi172.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2022-01-11T13:58:31.046 INFO:teuthology.orchestra.run.smithi172.stdout: python3-idna-2.5-5.el8.noarch 2022-01-11T13:58:31.047 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-6.2-6.el8.noarch 2022-01-11T13:58:31.047 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-01-11T13:58:31.048 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-01-11T13:58:31.048 INFO:teuthology.orchestra.run.smithi172.stdout: python3-lxml-4.2.3-3.el8.x86_64 2022-01-11T13:58:31.049 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako-1.0.6-13.el8.noarch 2022-01-11T13:58:31.049 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-01-11T13:58:31.049 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-01-11T13:58:31.050 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-01-11T13:58:31.050 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend-2.6-1.el8.noarch 2022-01-11T13:58:31.051 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-01-11T13:58:31.051 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser-2.14-14.el8.noarch 2022-01-11T13:58:31.052 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pysocks-1.6.8-3.el8.noarch 2022-01-11T13:58:31.052 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz-2017.2-9.el8.noarch 2022-01-11T13:58:31.052 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2022-01-11T13:58:31.053 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-01-11T13:58:31.053 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-01-11T13:58:31.054 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-01-11T13:58:31.054 INFO:teuthology.orchestra.run.smithi172.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-01-11T13:58:31.055 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3-1.24.2-5.el8.noarch 2022-01-11T13:58:31.055 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-01-11T13:58:31.056 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webencodings-0.5.1-6.el8.noarch 2022-01-11T13:58:31.056 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-01-11T13:58:31.057 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-01-11T13:58:31.057 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-01-11T13:58:31.057 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-01-11T13:58:31.057 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:31.058 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:58:31.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-base 2022-01-11T13:58:31.365 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 41/41 2022-01-11T13:58:31.366 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:31.366 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:58:31.366 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:58:31.366 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:58:31.367 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:58:31.367 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:58:31.367 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:58:31.367 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:58:31.368 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-01-11T13:58:31.368 INFO:teuthology.orchestra.run.smithi161.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-01-11T13:58:31.368 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-01-11T13:58:31.369 INFO:teuthology.orchestra.run.smithi161.stdout: python3-chardet-3.0.4-7.el8.noarch 2022-01-11T13:58:31.369 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-01-11T13:58:31.369 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-01-11T13:58:31.369 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-01-11T13:58:31.369 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cssselect-0.9.2-10.el8.noarch 2022-01-11T13:58:31.370 INFO:teuthology.orchestra.run.smithi161.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2022-01-11T13:58:31.370 INFO:teuthology.orchestra.run.smithi161.stdout: python3-idna-2.5-5.el8.noarch 2022-01-11T13:58:31.370 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-6.2-6.el8.noarch 2022-01-11T13:58:31.370 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-01-11T13:58:31.371 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-01-11T13:58:31.371 INFO:teuthology.orchestra.run.smithi161.stdout: python3-lxml-4.2.3-3.el8.x86_64 2022-01-11T13:58:31.371 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako-1.0.6-13.el8.noarch 2022-01-11T13:58:31.371 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-01-11T13:58:31.372 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-01-11T13:58:31.372 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-01-11T13:58:31.372 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend-2.6-1.el8.noarch 2022-01-11T13:58:31.372 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-01-11T13:58:31.373 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser-2.14-14.el8.noarch 2022-01-11T13:58:31.374 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pysocks-1.6.8-3.el8.noarch 2022-01-11T13:58:31.374 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz-2017.2-9.el8.noarch 2022-01-11T13:58:31.375 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2022-01-11T13:58:31.375 INFO:teuthology.orchestra.run.smithi161.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-01-11T13:58:31.375 INFO:teuthology.orchestra.run.smithi161.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-01-11T13:58:31.375 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-01-11T13:58:31.375 INFO:teuthology.orchestra.run.smithi161.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-01-11T13:58:31.376 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3-1.24.2-5.el8.noarch 2022-01-11T13:58:31.376 INFO:teuthology.orchestra.run.smithi161.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-01-11T13:58:31.376 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webencodings-0.5.1-6.el8.noarch 2022-01-11T13:58:31.376 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-01-11T13:58:31.376 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-01-11T13:58:31.377 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-01-11T13:58:31.377 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-01-11T13:58:31.377 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:31.378 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:58:31.548 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-base 2022-01-11T13:58:31.787 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:08 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:32.034 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:09 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:32.456 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-base-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:58:32.685 INFO:teuthology.orchestra.run.smithi161.stdout:Package ceph-base-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:58:32.695 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:58:32.697 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2022-01-11T13:58:32.697 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:58:32.756 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install cephadm 2022-01-11T13:58:32.922 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:58:32.923 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2022-01-11T13:58:32.923 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:58:32.978 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install cephadm 2022-01-11T13:58:33.238 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:10 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:33.455 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:10 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:34.118 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:58:34.119 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:58:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2022-01-11T13:58:34.120 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:58:34.120 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:58:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 70 k 2022-01-11T13:58:34.120 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:34.121 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:58:34.121 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:58:34.121 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2022-01-11T13:58:34.122 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:34.122 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 70 k 2022-01-11T13:58:34.122 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 314 k 2022-01-11T13:58:34.122 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:58:34.331 INFO:teuthology.orchestra.run.smithi172.stdout:cephadm-17.0.0-10004.g17f14252.el8.noarch.rpm 332 kB/s | 70 kB 00:00 2022-01-11T13:58:34.332 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:58:34.332 INFO:teuthology.orchestra.run.smithi172.stdout:Total 330 kB/s | 70 kB 00:00 2022-01-11T13:58:34.332 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:58:34.340 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:58:34.340 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:58:34.342 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:58:34.343 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:58:34.344 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:58:34.344 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2022-01-11T13:58:34.344 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:58:34.345 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:58:34.345 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 70 k 2022-01-11T13:58:34.345 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:34.345 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:58:34.346 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:58:34.346 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2022-01-11T13:58:34.346 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:34.346 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 70 k 2022-01-11T13:58:34.347 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 314 k 2022-01-11T13:58:34.347 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:58:34.347 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:58:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:58:34.558 INFO:teuthology.orchestra.run.smithi161.stdout:cephadm-17.0.0-10004.g17f14252.el8.noarch.rpm 328 kB/s | 70 kB 00:00 2022-01-11T13:58:34.559 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:58:34.559 INFO:teuthology.orchestra.run.smithi161.stdout:Total 326 kB/s | 70 kB 00:00 2022-01-11T13:58:34.559 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:58:34.567 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:58:34.568 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:58:34.569 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:58:34.573 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:58:34.615 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:58:36.620 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:17.0.0-10004.g17f14252.el8.noarch 1/1 2022-01-11T13:58:36.837 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephadm-2:17.0.0-10004.g17f14252.el8.noarch 1/1 2022-01-11T13:58:36.839 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:17.0.0-10004.g17f14252.el8.noarch 1/1 2022-01-11T13:58:37.023 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:17.0.0-10004.g17f14252.el8.noarch 1/1 2022-01-11T13:58:37.066 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephadm-2:17.0.0-10004.g17f14252.el8.noarch 1/1 2022-01-11T13:58:37.250 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:17.0.0-10004.g17f14252.el8.noarch 1/1 2022-01-11T13:58:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephadm-2:17.0.0-10004.g17f14252.el8.noarch 1/1 2022-01-11T13:58:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:37.297 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:58:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:58:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:37.297 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:58:37.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-immutable-object-cache 2022-01-11T13:58:37.542 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephadm-2:17.0.0-10004.g17f14252.el8.noarch 1/1 2022-01-11T13:58:37.543 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:37.543 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:58:37.543 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:58:37.544 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:37.544 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:58:37.719 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-immutable-object-cache 2022-01-11T13:58:37.907 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:14 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:38.208 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:15 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:38.791 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:58:38.792 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:58:38.792 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2022-01-11T13:58:38.792 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:58:38.793 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:58:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-10004.g17f14252.el8 ceph 169 k 2022-01-11T13:58:38.793 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:38.793 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:58:38.793 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:58:38.794 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2022-01-11T13:58:38.794 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:38.794 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 169 k 2022-01-11T13:58:38.794 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 567 k 2022-01-11T13:58:38.795 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:58:39.103 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-immutable-object-cache-17.0.0-10004.g17f14 545 kB/s | 169 kB 00:00 2022-01-11T13:58:39.103 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:58:39.104 INFO:teuthology.orchestra.run.smithi172.stdout:Total 542 kB/s | 169 kB 00:00 2022-01-11T13:58:39.104 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:58:39.116 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:58:39.117 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:58:39.119 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:58:39.120 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:58:39.120 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2022-01-11T13:58:39.120 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:58:39.120 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:58:39.121 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-10004.g17f14252.el8 ceph 169 k 2022-01-11T13:58:39.121 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:39.121 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:58:39.121 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:58:39.122 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2022-01-11T13:58:39.122 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:39.122 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 169 k 2022-01-11T13:58:39.122 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 567 k 2022-01-11T13:58:39.123 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:58:39.164 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:58:39.167 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:58:39.282 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:58:39.407 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-immutable-object-cache-17.0.0-10004.g17f14 590 kB/s | 169 kB 00:00 2022-01-11T13:58:39.408 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:58:39.408 INFO:teuthology.orchestra.run.smithi161.stdout:Total 587 kB/s | 169 kB 00:00 2022-01-11T13:58:39.409 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:58:39.421 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:58:39.422 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:58:39.469 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:58:39.472 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:58:39.485 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-10004.g17f14252 1/1 2022-01-11T13:58:39.580 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:58:39.747 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-10004.g17f14252 1/1 2022-01-11T13:58:39.842 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-10004.g17f14252 1/1 2022-01-11T13:58:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-10004.g17f14252 1/1 2022-01-11T13:58:40.435 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-10004.g17f14252 1/1 2022-01-11T13:58:40.436 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:40.436 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:58:40.436 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:58:40.436 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:40.437 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:58:40.575 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr 2022-01-11T13:58:40.724 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-10004.g17f14252 1/1 2022-01-11T13:58:40.724 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:40.725 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:58:40.725 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:58:40.725 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:40.725 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:58:41.060 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:18 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:42.384 INFO:teuthology.orchestra.run.smithi161.stdout:Package ceph-mgr-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:58:42.424 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr 2022-01-11T13:58:42.623 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:58:42.624 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2022-01-11T13:58:42.624 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:58:42.917 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:19 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:42.969 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-dashboard 2022-01-11T13:58:43.449 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:20 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:43.579 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-mgr-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:58:43.822 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:58:43.823 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2022-01-11T13:58:43.823 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:58:43.879 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-dashboard 2022-01-11T13:58:44.334 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:58:44.335 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:58:44.335 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2022-01-11T13:58:44.336 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:58:44.336 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:58:44.336 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 1.5 M 2022-01-11T13:58:44.336 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2022-01-11T13:58:44.337 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 24 k 2022-01-11T13:58:44.337 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 15 k 2022-01-11T13:58:44.337 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2022-01-11T13:58:44.337 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-01-11T13:58:44.338 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-01-11T13:58:44.338 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:44.338 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:58:44.338 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:58:44.339 INFO:teuthology.orchestra.run.smithi161.stdout:Install 6 Packages 2022-01-11T13:58:44.339 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:44.340 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 1.8 M 2022-01-11T13:58:44.340 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 7.4 M 2022-01-11T13:58:44.340 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:58:44.359 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:21 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:44.514 INFO:teuthology.orchestra.run.smithi161.stdout:(1/6): ceph-grafana-dashboards-17.0.0-10004.g17 135 kB/s | 24 kB 00:00 2022-01-11T13:58:44.538 INFO:teuthology.orchestra.run.smithi161.stdout:(2/6): ceph-prometheus-alerts-17.0.0-10004.g17f 73 kB/s | 15 kB 00:00 2022-01-11T13:58:44.544 INFO:teuthology.orchestra.run.smithi161.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2022-01-11T13:58:44.622 INFO:teuthology.orchestra.run.smithi161.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 399 kB/s | 34 kB 00:00 2022-01-11T13:58:44.683 INFO:teuthology.orchestra.run.smithi161.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2022-01-11T13:58:44.704 INFO:teuthology.orchestra.run.smithi161.stdout:(6/6): ceph-mgr-dashboard-17.0.0-10004.g17f1425 4.2 MB/s | 1.5 MB 00:00 2022-01-11T13:58:44.707 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:58:44.708 INFO:teuthology.orchestra.run.smithi161.stdout:Total 4.9 MB/s | 1.8 MB 00:00 2022-01-11T13:58:44.710 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:58:44.721 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:58:44.721 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:58:44.780 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:58:44.784 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:58:45.028 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:58:45.248 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:58:45.249 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:58:45.249 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2022-01-11T13:58:45.250 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:58:45.250 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:58:45.250 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 1.5 M 2022-01-11T13:58:45.250 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2022-01-11T13:58:45.251 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 24 k 2022-01-11T13:58:45.251 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 15 k 2022-01-11T13:58:45.251 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2022-01-11T13:58:45.251 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-01-11T13:58:45.251 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-01-11T13:58:45.252 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:45.252 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:58:45.252 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:58:45.252 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2022-01-11T13:58:45.253 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:45.253 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 1.8 M 2022-01-11T13:58:45.254 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 7.4 M 2022-01-11T13:58:45.254 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:58:45.297 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-01-11T13:58:45.427 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): ceph-grafana-dashboards-17.0.0-10004.g17 135 kB/s | 24 kB 00:00 2022-01-11T13:58:45.435 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): ceph-prometheus-alerts-17.0.0-10004.g17f 80 kB/s | 15 kB 00:00 2022-01-11T13:58:45.457 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2022-01-11T13:58:45.521 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 392 kB/s | 34 kB 00:00 2022-01-11T13:58:45.523 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-01-11T13:58:45.596 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2022-01-11T13:58:45.636 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): ceph-mgr-dashboard-17.0.0-10004.g17f1425 4.0 MB/s | 1.5 MB 00:00 2022-01-11T13:58:45.638 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:58:45.639 INFO:teuthology.orchestra.run.smithi172.stdout:Total 4.7 MB/s | 1.8 MB 00:00 2022-01-11T13:58:45.641 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:58:45.649 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:58:45.649 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:58:45.713 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:58:45.716 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:58:45.757 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-01-11T13:58:45.903 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-10004.g17f14252.el8. 4/6 2022-01-11T13:58:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:58:46.259 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-01-11T13:58:46.290 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-10004.g17f14252.el8 5/6 2022-01-11T13:58:46.503 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-01-11T13:58:46.533 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-10004.g17f14252.el8.noar 6/6 2022-01-11T13:58:46.736 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-01-11T13:58:46.774 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-10004.g17f14252.el8.noar 6/6 2022-01-11T13:58:46.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-10004.g17f14252.el8 1/6 2022-01-11T13:58:46.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-10004.g17f14252.el8.noar 2/6 2022-01-11T13:58:46.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-10004.g17f14252.el8. 3/6 2022-01-11T13:58:46.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-01-11T13:58:46.776 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-01-11T13:58:46.884 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-10004.g17f14252.el8. 4/6 2022-01-11T13:58:47.058 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-01-11T13:58:47.058 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:47.059 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:58:47.059 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:58:47.059 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:58:47.059 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:58:47.060 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-01-11T13:58:47.060 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-01-11T13:58:47.060 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes-2.4.1-12.el8.noarch 2022-01-11T13:58:47.060 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:47.061 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:58:47.134 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-diskprediction-local 2022-01-11T13:58:47.279 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-10004.g17f14252.el8 5/6 2022-01-11T13:58:47.522 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-10004.g17f14252.el8.noar 6/6 2022-01-11T13:58:47.608 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:24 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-10004.g17f14252.el8.noar 6/6 2022-01-11T13:58:47.761 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-10004.g17f14252.el8 1/6 2022-01-11T13:58:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-10004.g17f14252.el8.noar 2/6 2022-01-11T13:58:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-10004.g17f14252.el8. 3/6 2022-01-11T13:58:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-01-11T13:58:47.762 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-01-11T13:58:48.072 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-01-11T13:58:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:48.073 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:58:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:58:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:58:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:58:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-01-11T13:58:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-01-11T13:58:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes-2.4.1-12.el8.noarch 2022-01-11T13:58:48.075 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:48.075 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:58:48.153 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-diskprediction-local 2022-01-11T13:58:48.486 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:58:48.487 INFO:teuthology.orchestra.run.smithi161.stdout:===================================================================================================== 2022-01-11T13:58:48.487 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2022-01-11T13:58:48.487 INFO:teuthology.orchestra.run.smithi161.stdout:===================================================================================================== 2022-01-11T13:58:48.488 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:58:48.488 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 7.4 M 2022-01-11T13:58:48.488 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrading: 2022-01-11T13:58:48.488 INFO:teuthology.orchestra.run.smithi161.stdout: chkconfig x86_64 1.19.1-1.el8 CentOS-Base 198 k 2022-01-11T13:58:48.489 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2022-01-11T13:58:48.489 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran x86_64 8.5.0-4.el8_5 CentOS-Base 643 k 2022-01-11T13:58:48.489 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath x86_64 8.5.0-4.el8_5 CentOS-Base 170 k 2022-01-11T13:58:48.489 INFO:teuthology.orchestra.run.smithi161.stdout: openblas x86_64 0.3.12-1.el8 CentOS-AppStream 4.6 M 2022-01-11T13:58:48.489 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-threads x86_64 0.3.12-1.el8 CentOS-AppStream 4.7 M 2022-01-11T13:58:48.490 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-01-11T13:58:48.490 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-01-11T13:58:48.490 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-01-11T13:58:48.490 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:48.491 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:58:48.491 INFO:teuthology.orchestra.run.smithi161.stdout:===================================================================================================== 2022-01-11T13:58:48.491 INFO:teuthology.orchestra.run.smithi161.stdout:Install 8 Packages 2022-01-11T13:58:48.491 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrade 1 Package 2022-01-11T13:58:48.492 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:48.492 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 35 M 2022-01-11T13:58:48.492 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:58:48.611 INFO:teuthology.orchestra.run.smithi161.stdout:(1/9): openblas-threads-0.3.12-1.el8.x86_64.rpm 39 MB/s | 4.7 MB 00:00 2022-01-11T13:58:48.626 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:25 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:48.664 INFO:teuthology.orchestra.run.smithi161.stdout:(2/9): python3-numpy-1.14.3-10.el8.x86_64.rpm 72 MB/s | 3.7 MB 00:00 2022-01-11T13:58:48.669 INFO:teuthology.orchestra.run.smithi161.stdout:(3/9): python3-numpy-f2py-1.14.3-10.el8.x86_64. 44 MB/s | 225 kB 00:00 2022-01-11T13:58:48.754 INFO:teuthology.orchestra.run.smithi161.stdout:(4/9): openblas-0.3.12-1.el8.x86_64.rpm 18 MB/s | 4.6 MB 00:00 2022-01-11T13:58:48.784 INFO:teuthology.orchestra.run.smithi161.stdout:(5/9): libgfortran-8.5.0-4.el8_5.x86_64.rpm 19 MB/s | 643 kB 00:00 2022-01-11T13:58:48.794 INFO:teuthology.orchestra.run.smithi161.stdout:(6/9): libquadmath-8.5.0-4.el8_5.x86_64.rpm 17 MB/s | 170 kB 00:00 2022-01-11T13:58:48.812 INFO:teuthology.orchestra.run.smithi161.stdout:(7/9): chkconfig-1.19.1-1.el8.x86_64.rpm 11 MB/s | 198 kB 00:00 2022-01-11T13:58:48.927 INFO:teuthology.orchestra.run.smithi161.stdout:(8/9): python3-scipy-1.0.0-21.module_el8.5.0+77 53 MB/s | 14 MB 00:00 2022-01-11T13:58:49.110 INFO:teuthology.orchestra.run.smithi161.stdout:(9/9): ceph-mgr-diskprediction-local-17.0.0-100 12 MB/s | 7.4 MB 00:00 2022-01-11T13:58:49.113 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:58:49.114 INFO:teuthology.orchestra.run.smithi161.stdout:Total 56 MB/s | 35 MB 00:00 2022-01-11T13:58:49.129 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:58:49.159 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:58:49.159 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:58:49.529 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:58:49.532 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:58:49.532 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:58:49.533 INFO:teuthology.orchestra.run.smithi172.stdout:===================================================================================================== 2022-01-11T13:58:49.533 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2022-01-11T13:58:49.534 INFO:teuthology.orchestra.run.smithi172.stdout:===================================================================================================== 2022-01-11T13:58:49.534 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:58:49.534 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 7.4 M 2022-01-11T13:58:49.534 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrading: 2022-01-11T13:58:49.535 INFO:teuthology.orchestra.run.smithi172.stdout: chkconfig x86_64 1.19.1-1.el8 CentOS-Base 198 k 2022-01-11T13:58:49.535 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2022-01-11T13:58:49.535 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran x86_64 8.5.0-4.el8_5 CentOS-Base 643 k 2022-01-11T13:58:49.535 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath x86_64 8.5.0-4.el8_5 CentOS-Base 170 k 2022-01-11T13:58:49.536 INFO:teuthology.orchestra.run.smithi172.stdout: openblas x86_64 0.3.12-1.el8 CentOS-AppStream 4.6 M 2022-01-11T13:58:49.536 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-threads x86_64 0.3.12-1.el8 CentOS-AppStream 4.7 M 2022-01-11T13:58:49.536 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-01-11T13:58:49.536 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-01-11T13:58:49.536 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-01-11T13:58:49.537 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:49.537 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:58:49.537 INFO:teuthology.orchestra.run.smithi172.stdout:===================================================================================================== 2022-01-11T13:58:49.537 INFO:teuthology.orchestra.run.smithi172.stdout:Install 8 Packages 2022-01-11T13:58:49.538 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrade 1 Package 2022-01-11T13:58:49.538 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:58:49.538 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 35 M 2022-01-11T13:58:49.539 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:58:49.635 INFO:teuthology.orchestra.run.smithi172.stdout:(1/9): openblas-0.3.12-1.el8.x86_64.rpm 46 MB/s | 4.6 MB 00:00 2022-01-11T13:58:49.690 INFO:teuthology.orchestra.run.smithi172.stdout:(2/9): python3-numpy-1.14.3-10.el8.x86_64.rpm 67 MB/s | 3.7 MB 00:00 2022-01-11T13:58:49.696 INFO:teuthology.orchestra.run.smithi172.stdout:(3/9): python3-numpy-f2py-1.14.3-10.el8.x86_64. 42 MB/s | 225 kB 00:00 2022-01-11T13:58:49.763 INFO:teuthology.orchestra.run.smithi172.stdout:(4/9): openblas-threads-0.3.12-1.el8.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2022-01-11T13:58:49.792 INFO:teuthology.orchestra.run.smithi172.stdout:(5/9): libgfortran-8.5.0-4.el8_5.x86_64.rpm 22 MB/s | 643 kB 00:00 2022-01-11T13:58:49.801 INFO:teuthology.orchestra.run.smithi172.stdout:(6/9): libquadmath-8.5.0-4.el8_5.x86_64.rpm 19 MB/s | 170 kB 00:00 2022-01-11T13:58:49.815 INFO:teuthology.orchestra.run.smithi172.stdout:(7/9): chkconfig-1.19.1-1.el8.x86_64.rpm 14 MB/s | 198 kB 00:00 2022-01-11T13:58:49.950 INFO:teuthology.orchestra.run.smithi172.stdout:(8/9): python3-scipy-1.0.0-21.module_el8.5.0+77 54 MB/s | 14 MB 00:00 2022-01-11T13:58:50.134 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:58:50.218 INFO:teuthology.orchestra.run.smithi172.stdout:(9/9): ceph-mgr-diskprediction-local-17.0.0-100 11 MB/s | 7.4 MB 00:00 2022-01-11T13:58:50.221 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:58:50.222 INFO:teuthology.orchestra.run.smithi172.stdout:Total 51 MB/s | 35 MB 00:00 2022-01-11T13:58:50.237 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:58:50.266 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:58:50.267 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:58:50.278 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libquadmath-8.5.0-4.el8_5.x86_64 1/10 2022-01-11T13:58:50.395 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libquadmath-8.5.0-4.el8_5.x86_64 1/10 2022-01-11T13:58:50.612 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libgfortran-8.5.0-4.el8_5.x86_64 2/10 2022-01-11T13:58:50.650 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:58:50.653 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:58:51.152 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libgfortran-8.5.0-4.el8_5.x86_64 2/10 2022-01-11T13:58:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:58:51.328 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-0.3.12-1.el8.x86_64 3/10 2022-01-11T13:58:51.408 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libquadmath-8.5.0-4.el8_5.x86_64 1/10 2022-01-11T13:58:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libquadmath-8.5.0-4.el8_5.x86_64 1/10 2022-01-11T13:58:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgfortran-8.5.0-4.el8_5.x86_64 2/10 2022-01-11T13:58:51.892 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 3/10 2022-01-11T13:58:52.020 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 4/10 2022-01-11T13:58:52.296 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libgfortran-8.5.0-4.el8_5.x86_64 2/10 2022-01-11T13:58:52.458 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-0.3.12-1.el8.x86_64 3/10 2022-01-11T13:58:52.708 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 4/10 2022-01-11T13:58:53.003 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 3/10 2022-01-11T13:58:53.075 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/10 2022-01-11T13:58:53.266 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 4/10 2022-01-11T13:58:53.771 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : chkconfig-1.19.1-1.el8.x86_64 6/10 2022-01-11T13:58:53.981 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 4/10 2022-01-11T13:58:54.019 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 7/10 2022-01-11T13:58:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/10 2022-01-11T13:58:54.766 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : chkconfig-1.19.1-1.el8.x86_64 6/10 2022-01-11T13:58:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 7/10 2022-01-11T13:58:55.833 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 7/10 2022-01-11T13:58:56.782 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 7/10 2022-01-11T13:58:57.051 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225 8/10 2022-01-11T13:58:57.261 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-10004.g17f1 9/10 2022-01-11T13:58:57.276 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-10004.g17f1 9/10 2022-01-11T13:58:57.443 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : chkconfig-1.13-2.el8.x86_64 10/10 2022-01-11T13:58:57.844 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: chkconfig-1.13-2.el8.x86_64 10/10 2022-01-11T13:58:57.845 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-10004.g17f1 1/10 2022-01-11T13:58:57.845 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 2/10 2022-01-11T13:58:57.845 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 3/10 2022-01-11T13:58:57.846 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/10 2022-01-11T13:58:57.846 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/10 2022-01-11T13:58:57.846 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225 6/10 2022-01-11T13:58:57.846 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libgfortran-8.5.0-4.el8_5.x86_64 7/10 2022-01-11T13:58:57.846 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libquadmath-8.5.0-4.el8_5.x86_64 8/10 2022-01-11T13:58:57.847 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : chkconfig-1.19.1-1.el8.x86_64 9/10 2022-01-11T13:58:58.013 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225 8/10 2022-01-11T13:58:58.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : chkconfig-1.13-2.el8.x86_64 10/10 2022-01-11T13:58:58.240 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:58.240 INFO:teuthology.orchestra.run.smithi161.stdout:Upgraded: 2022-01-11T13:58:58.241 INFO:teuthology.orchestra.run.smithi161.stdout: chkconfig-1.19.1-1.el8.x86_64 2022-01-11T13:58:58.241 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:58.241 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:58:58.241 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:58:58.242 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran-8.5.0-4.el8_5.x86_64 2022-01-11T13:58:58.242 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath-8.5.0-4.el8_5.x86_64 2022-01-11T13:58:58.242 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-0.3.12-1.el8.x86_64 2022-01-11T13:58:58.242 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-threads-0.3.12-1.el8.x86_64 2022-01-11T13:58:58.243 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-01-11T13:58:58.243 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-01-11T13:58:58.243 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-01-11T13:58:58.243 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:58:58.243 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:58:58.425 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-rook 2022-01-11T13:58:58.907 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:35 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:58:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-10004.g17f1 9/10 2022-01-11T13:58:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-10004.g17f1 9/10 2022-01-11T13:58:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : chkconfig-1.13-2.el8.x86_64 10/10 2022-01-11T13:58:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: chkconfig-1.13-2.el8.x86_64 10/10 2022-01-11T13:58:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-10004.g17f1 1/10 2022-01-11T13:58:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 2/10 2022-01-11T13:58:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 3/10 2022-01-11T13:58:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/10 2022-01-11T13:58:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/10 2022-01-11T13:58:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225 6/10 2022-01-11T13:58:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgfortran-8.5.0-4.el8_5.x86_64 7/10 2022-01-11T13:58:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libquadmath-8.5.0-4.el8_5.x86_64 8/10 2022-01-11T13:58:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : chkconfig-1.19.1-1.el8.x86_64 9/10 2022-01-11T13:59:00.028 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : chkconfig-1.13-2.el8.x86_64 10/10 2022-01-11T13:59:00.029 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:00.029 INFO:teuthology.orchestra.run.smithi172.stdout:Upgraded: 2022-01-11T13:59:00.029 INFO:teuthology.orchestra.run.smithi172.stdout: chkconfig-1.19.1-1.el8.x86_64 2022-01-11T13:59:00.029 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:00.029 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:59:00.030 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:59:00.030 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran-8.5.0-4.el8_5.x86_64 2022-01-11T13:59:00.030 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath-8.5.0-4.el8_5.x86_64 2022-01-11T13:59:00.031 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-0.3.12-1.el8.x86_64 2022-01-11T13:59:00.031 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-threads-0.3.12-1.el8.x86_64 2022-01-11T13:59:00.031 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-01-11T13:59:00.031 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-01-11T13:59:00.031 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-01-11T13:59:00.032 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:00.032 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:00.118 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:00.120 INFO:teuthology.orchestra.run.smithi161.stdout:========================================================================================= 2022-01-11T13:59:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2022-01-11T13:59:00.120 INFO:teuthology.orchestra.run.smithi161.stdout:========================================================================================= 2022-01-11T13:59:00.120 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:59:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 47 k 2022-01-11T13:59:00.121 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2022-01-11T13:59:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-01-11T13:59:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-01-11T13:59:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-01-11T13:59:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-01-11T13:59:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-01-11T13:59:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-01-11T13:59:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2022-01-11T13:59:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-01-11T13:59:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-01-11T13:59:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2022-01-11T13:59:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-01-11T13:59:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-01-11T13:59:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:00.124 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:59:00.124 INFO:teuthology.orchestra.run.smithi161.stdout:========================================================================================= 2022-01-11T13:59:00.124 INFO:teuthology.orchestra.run.smithi161.stdout:Install 13 Packages 2022-01-11T13:59:00.125 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:00.125 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 1.9 M 2022-01-11T13:59:00.126 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 25 M 2022-01-11T13:59:00.126 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:59:00.155 INFO:teuthology.orchestra.run.smithi161.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 830 kB/s | 27 kB 00:00 2022-01-11T13:59:00.162 INFO:teuthology.orchestra.run.smithi161.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 508 kB/s | 20 kB 00:00 2022-01-11T13:59:00.163 INFO:teuthology.orchestra.run.smithi161.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 16 MB/s | 126 kB 00:00 2022-01-11T13:59:00.165 INFO:teuthology.orchestra.run.smithi161.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 35 MB/s | 110 kB 00:00 2022-01-11T13:59:00.166 INFO:teuthology.orchestra.run.smithi161.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 42 MB/s | 155 kB 00:00 2022-01-11T13:59:00.167 INFO:teuthology.orchestra.run.smithi161.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 27 MB/s | 43 kB 00:00 2022-01-11T13:59:00.243 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-rook 2022-01-11T13:59:00.266 INFO:teuthology.orchestra.run.smithi161.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 331 kB/s | 33 kB 00:00 2022-01-11T13:59:00.269 INFO:teuthology.orchestra.run.smithi161.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 160 kB/s | 16 kB 00:00 2022-01-11T13:59:00.331 INFO:teuthology.orchestra.run.smithi161.stdout:(9/13): ceph-mgr-rook-17.0.0-10004.g17f14252.el 226 kB/s | 47 kB 00:00 2022-01-11T13:59:00.337 INFO:teuthology.orchestra.run.smithi161.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 1.1 MB/s | 81 kB 00:00 2022-01-11T13:59:00.366 INFO:teuthology.orchestra.run.smithi161.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 2.1 MB/s | 61 kB 00:00 2022-01-11T13:59:00.447 INFO:teuthology.orchestra.run.smithi161.stdout:(12/13): python3-rsa-4.8-1.el8.noarch.rpm 543 kB/s | 61 kB 00:00 2022-01-11T13:59:00.489 INFO:teuthology.orchestra.run.smithi161.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.3 MB/s | 1.2 MB 00:00 2022-01-11T13:59:00.491 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:00.491 INFO:teuthology.orchestra.run.smithi161.stdout:Total 5.2 MB/s | 1.9 MB 00:00 2022-01-11T13:59:00.501 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:59:00.515 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:59:00.515 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:59:00.590 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:59:00.593 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:59:00.732 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:37 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:01.089 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:59:01.323 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-01-11T13:59:01.487 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-01-11T13:59:01.619 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:01.622 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================= 2022-01-11T13:59:01.623 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2022-01-11T13:59:01.623 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================= 2022-01-11T13:59:01.623 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:59:01.624 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 47 k 2022-01-11T13:59:01.624 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2022-01-11T13:59:01.624 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-01-11T13:59:01.624 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-01-11T13:59:01.625 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-01-11T13:59:01.625 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-01-11T13:59:01.625 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-01-11T13:59:01.625 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-01-11T13:59:01.626 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2022-01-11T13:59:01.626 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-01-11T13:59:01.626 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-01-11T13:59:01.626 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2022-01-11T13:59:01.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-01-11T13:59:01.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-01-11T13:59:01.627 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:01.627 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:59:01.627 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================= 2022-01-11T13:59:01.628 INFO:teuthology.orchestra.run.smithi172.stdout:Install 13 Packages 2022-01-11T13:59:01.628 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:01.629 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 1.9 M 2022-01-11T13:59:01.629 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 25 M 2022-01-11T13:59:01.629 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:59:01.655 INFO:teuthology.orchestra.run.smithi172.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 872 kB/s | 27 kB 00:00 2022-01-11T13:59:01.658 INFO:teuthology.orchestra.run.smithi172.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 34 MB/s | 126 kB 00:00 2022-01-11T13:59:01.659 INFO:teuthology.orchestra.run.smithi172.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 575 kB/s | 20 kB 00:00 2022-01-11T13:59:01.662 INFO:teuthology.orchestra.run.smithi172.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 34 MB/s | 110 kB 00:00 2022-01-11T13:59:01.663 INFO:teuthology.orchestra.run.smithi172.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 42 MB/s | 155 kB 00:00 2022-01-11T13:59:01.669 INFO:teuthology.orchestra.run.smithi172.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 6.8 MB/s | 43 kB 00:00 2022-01-11T13:59:01.722 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-01-11T13:59:01.751 INFO:teuthology.orchestra.run.smithi172.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 197 kB/s | 16 kB 00:00 2022-01-11T13:59:01.753 INFO:teuthology.orchestra.run.smithi172.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 385 kB/s | 33 kB 00:00 2022-01-11T13:59:01.833 INFO:teuthology.orchestra.run.smithi172.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 989 kB/s | 81 kB 00:00 2022-01-11T13:59:01.855 INFO:teuthology.orchestra.run.smithi172.stdout:(10/13): ceph-mgr-rook-17.0.0-10004.g17f14252.e 204 kB/s | 47 kB 00:00 2022-01-11T13:59:01.861 INFO:teuthology.orchestra.run.smithi172.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 2.1 MB/s | 61 kB 00:00 2022-01-11T13:59:01.927 INFO:teuthology.orchestra.run.smithi172.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 6.7 MB/s | 1.2 MB 00:00 2022-01-11T13:59:01.959 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-01-11T13:59:01.968 INFO:teuthology.orchestra.run.smithi172.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 539 kB/s | 61 kB 00:00 2022-01-11T13:59:01.970 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:01.971 INFO:teuthology.orchestra.run.smithi172.stdout:Total 5.5 MB/s | 1.9 MB 00:00 2022-01-11T13:59:01.981 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:59:01.995 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:59:01.996 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:59:02.073 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:59:02.076 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:59:02.167 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-01-11T13:59:02.401 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-01-11T13:59:02.607 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:59:02.728 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-01-11T13:59:02.844 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-01-11T13:59:02.941 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-01-11T13:59:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-01-11T13:59:03.309 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-01-11T13:59:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-01-11T13:59:03.763 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-01-11T13:59:03.996 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-01-11T13:59:04.075 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-01-11T13:59:04.346 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-01-11T13:59:04.534 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-01-11T13:59:04.728 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-01-11T13:59:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-01-11T13:59:04.951 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-rook-2:17.0.0-10004.g17f14252.el8.noarch 13/13 2022-01-11T13:59:04.954 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-01-11T13:59:05.208 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-10004.g17f14252.el8.noarch 13/13 2022-01-11T13:59:05.208 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-rook-2:17.0.0-10004.g17f14252.el8.noarch 1/13 2022-01-11T13:59:05.208 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-01-11T13:59:05.209 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-01-11T13:59:05.209 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-01-11T13:59:05.209 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-01-11T13:59:05.209 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-01-11T13:59:05.209 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-01-11T13:59:05.210 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-01-11T13:59:05.210 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-01-11T13:59:05.210 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-01-11T13:59:05.210 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-01-11T13:59:05.211 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-01-11T13:59:05.863 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-01-11T13:59:05.864 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:05.864 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:59:05.864 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:59:05.864 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-01-11T13:59:05.865 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-01-11T13:59:05.865 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-01-11T13:59:05.865 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-01-11T13:59:05.865 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-01-11T13:59:05.866 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-01-11T13:59:05.867 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-01-11T13:59:05.867 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-01-11T13:59:05.867 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-01-11T13:59:05.867 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-01-11T13:59:05.868 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa-4.8-1.el8.noarch 2022-01-11T13:59:05.868 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-01-11T13:59:05.868 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:05.869 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:05.948 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-cephadm 2022-01-11T13:59:06.088 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-01-11T13:59:06.346 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-01-11T13:59:06.426 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:43 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:06.571 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-01-11T13:59:06.774 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-01-11T13:59:07.030 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-rook-2:17.0.0-10004.g17f14252.el8.noarch 13/13 2022-01-11T13:59:07.285 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-10004.g17f14252.el8.noarch 13/13 2022-01-11T13:59:07.285 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-rook-2:17.0.0-10004.g17f14252.el8.noarch 1/13 2022-01-11T13:59:07.285 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-01-11T13:59:07.286 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-01-11T13:59:07.286 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-01-11T13:59:07.286 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-01-11T13:59:07.286 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-01-11T13:59:07.287 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-01-11T13:59:07.287 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-01-11T13:59:07.287 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-01-11T13:59:07.287 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-01-11T13:59:07.288 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-01-11T13:59:07.288 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-01-11T13:59:07.320 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:07.321 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:07.321 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2022-01-11T13:59:07.322 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:07.322 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:59:07.322 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 102 k 2022-01-11T13:59:07.322 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2022-01-11T13:59:07.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-01-11T13:59:07.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-01-11T13:59:07.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-01-11T13:59:07.323 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:07.323 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:59:07.324 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:07.324 INFO:teuthology.orchestra.run.smithi161.stdout:Install 4 Packages 2022-01-11T13:59:07.324 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:07.325 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 5.9 M 2022-01-11T13:59:07.325 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 25 M 2022-01-11T13:59:07.325 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:59:07.376 INFO:teuthology.orchestra.run.smithi161.stdout:(1/4): python3-jinja2-2.10.1-3.el8.noarch.rpm 9.9 MB/s | 538 kB 00:00 2022-01-11T13:59:07.418 INFO:teuthology.orchestra.run.smithi161.stdout:(2/4): python3-babel-2.5.1-7.el8.noarch.rpm 51 MB/s | 4.8 MB 00:00 2022-01-11T13:59:07.666 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-01-11T13:59:07.666 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:07.667 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:59:07.667 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:59:07.667 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-01-11T13:59:07.667 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-01-11T13:59:07.668 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-01-11T13:59:07.668 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-01-11T13:59:07.668 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-01-11T13:59:07.668 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-01-11T13:59:07.669 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-01-11T13:59:07.669 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-01-11T13:59:07.669 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-01-11T13:59:07.669 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-01-11T13:59:07.669 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa-4.8-1.el8.noarch 2022-01-11T13:59:07.670 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-01-11T13:59:07.670 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:07.670 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:07.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-cephadm 2022-01-11T13:59:07.808 INFO:teuthology.orchestra.run.smithi161.stdout:(3/4): ceph-mgr-cephadm-17.0.0-10004.g17f14252. 210 kB/s | 102 kB 00:00 2022-01-11T13:59:08.173 INFO:teuthology.orchestra.run.smithi161.stdout:(4/4): python3-asyncssh-2.7.0-2.el8.noarch.rpm 598 kB/s | 476 kB 00:00 2022-01-11T13:59:08.176 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:08.177 INFO:teuthology.orchestra.run.smithi161.stdout:Total 6.9 MB/s | 5.9 MB 00:00 2022-01-11T13:59:08.182 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:59:08.192 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:59:08.193 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:59:08.244 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:45 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:08.279 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:59:08.282 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:59:08.640 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:59:09.126 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:09.127 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:09.127 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2022-01-11T13:59:09.127 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:09.128 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:59:09.128 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 102 k 2022-01-11T13:59:09.128 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2022-01-11T13:59:09.128 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-01-11T13:59:09.129 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-01-11T13:59:09.129 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-01-11T13:59:09.129 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:09.129 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:59:09.130 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:09.130 INFO:teuthology.orchestra.run.smithi172.stdout:Install 4 Packages 2022-01-11T13:59:09.130 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:09.130 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 5.9 M 2022-01-11T13:59:09.131 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 25 M 2022-01-11T13:59:09.131 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:59:09.191 INFO:teuthology.orchestra.run.smithi172.stdout:(1/4): python3-jinja2-2.10.1-3.el8.noarch.rpm 8.4 MB/s | 538 kB 00:00 2022-01-11T13:59:09.240 INFO:teuthology.orchestra.run.smithi172.stdout:(2/4): python3-babel-2.5.1-7.el8.noarch.rpm 43 MB/s | 4.8 MB 00:00 2022-01-11T13:59:09.477 INFO:teuthology.orchestra.run.smithi172.stdout:(3/4): ceph-mgr-cephadm-17.0.0-10004.g17f14252. 293 kB/s | 102 kB 00:00 2022-01-11T13:59:09.626 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 1/4 2022-01-11T13:59:09.649 INFO:teuthology.orchestra.run.smithi172.stdout:(4/4): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.0 MB/s | 476 kB 00:00 2022-01-11T13:59:09.652 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:09.652 INFO:teuthology.orchestra.run.smithi172.stdout:Total 11 MB/s | 5.9 MB 00:00 2022-01-11T13:59:09.657 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:59:09.666 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:59:09.666 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:59:09.752 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:59:09.755 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:59:10.019 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 2/4 2022-01-11T13:59:10.267 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 3/4 2022-01-11T13:59:10.302 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:59:10.483 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-10004.g17f14252.el8.noarch 4/4 2022-01-11T13:59:10.733 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-10004.g17f14252.el8.noarch 4/4 2022-01-11T13:59:10.733 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-10004.g17f14252.el8.noarch 1/4 2022-01-11T13:59:10.733 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/4 2022-01-11T13:59:10.734 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/4 2022-01-11T13:59:11.220 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/4 2022-01-11T13:59:11.220 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:11.221 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:59:11.221 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:59:11.221 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-01-11T13:59:11.221 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel-2.5.1-7.el8.noarch 2022-01-11T13:59:11.221 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-01-11T13:59:11.222 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:11.222 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:11.288 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-fuse 2022-01-11T13:59:11.301 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 1/4 2022-01-11T13:59:11.692 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 2/4 2022-01-11T13:59:11.768 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:48 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:11.946 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 3/4 2022-01-11T13:59:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-10004.g17f14252.el8.noarch 4/4 2022-01-11T13:59:12.410 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-10004.g17f14252.el8.noarch 4/4 2022-01-11T13:59:12.410 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-10004.g17f14252.el8.noarch 1/4 2022-01-11T13:59:12.411 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/4 2022-01-11T13:59:12.411 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/4 2022-01-11T13:59:12.651 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:12.652 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:12.652 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2022-01-11T13:59:12.653 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:12.653 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:59:12.653 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse x86_64 2:17.0.0-10004.g17f14252.el8 ceph 871 k 2022-01-11T13:59:12.653 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2022-01-11T13:59:12.653 INFO:teuthology.orchestra.run.smithi161.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2022-01-11T13:59:12.654 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:12.654 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:59:12.654 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:12.654 INFO:teuthology.orchestra.run.smithi161.stdout:Install 2 Packages 2022-01-11T13:59:12.655 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:12.655 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 953 k 2022-01-11T13:59:12.655 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 3.1 M 2022-01-11T13:59:12.655 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:59:12.684 INFO:teuthology.orchestra.run.smithi161.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.6 MB/s | 82 kB 00:00 2022-01-11T13:59:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/4 2022-01-11T13:59:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:12.708 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:59:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:59:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-01-11T13:59:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel-2.5.1-7.el8.noarch 2022-01-11T13:59:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-01-11T13:59:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:12.710 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:12.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-fuse 2022-01-11T13:59:13.139 INFO:teuthology.orchestra.run.smithi161.stdout:(2/2): ceph-fuse-17.0.0-10004.g17f14252.el8.x86 1.8 MB/s | 871 kB 00:00 2022-01-11T13:59:13.139 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:13.140 INFO:teuthology.orchestra.run.smithi161.stdout:Total 1.9 MB/s | 953 kB 00:00 2022-01-11T13:59:13.141 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:59:13.163 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:59:13.163 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:59:13.242 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:59:13.244 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:59:13.274 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:50 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:13.410 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:59:13.734 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2022-01-11T13:59:13.924 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 2/2 2022-01-11T13:59:14.188 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:14.189 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:14.190 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2022-01-11T13:59:14.190 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:14.190 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:59:14.190 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse x86_64 2:17.0.0-10004.g17f14252.el8 ceph 871 k 2022-01-11T13:59:14.191 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2022-01-11T13:59:14.191 INFO:teuthology.orchestra.run.smithi172.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2022-01-11T13:59:14.191 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:14.191 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:59:14.192 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:14.192 INFO:teuthology.orchestra.run.smithi172.stdout:Install 2 Packages 2022-01-11T13:59:14.192 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:14.192 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 953 k 2022-01-11T13:59:14.193 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 3.1 M 2022-01-11T13:59:14.193 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:59:14.222 INFO:teuthology.orchestra.run.smithi172.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.5 MB/s | 82 kB 00:00 2022-01-11T13:59:14.266 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 2/2 2022-01-11T13:59:14.266 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 1/2 2022-01-11T13:59:14.537 INFO:teuthology.orchestra.run.smithi172.stdout:(2/2): ceph-fuse-17.0.0-10004.g17f14252.el8.x86 2.5 MB/s | 871 kB 00:00 2022-01-11T13:59:14.538 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:14.539 INFO:teuthology.orchestra.run.smithi172.stdout:Total 2.7 MB/s | 953 kB 00:00 2022-01-11T13:59:14.541 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:59:14.563 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:59:14.563 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:59:14.565 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2022-01-11T13:59:14.565 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:14.566 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:59:14.566 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2022-01-11T13:59:14.567 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:14.567 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:14.641 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:59:14.644 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:59:14.698 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-volume 2022-01-11T13:59:14.802 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:59:15.063 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2022-01-11T13:59:15.183 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:52 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:15.236 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 2/2 2022-01-11T13:59:15.622 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 2/2 2022-01-11T13:59:15.623 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 1/2 2022-01-11T13:59:15.955 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2022-01-11T13:59:15.956 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:15.956 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:59:15.956 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2022-01-11T13:59:15.956 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:15.957 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:16.057 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:16.057 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:16.057 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2022-01-11T13:59:16.058 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:16.058 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:59:16.058 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 256 k 2022-01-11T13:59:16.058 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:16.059 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:59:16.059 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:16.059 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2022-01-11T13:59:16.059 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:16.060 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 256 k 2022-01-11T13:59:16.060 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 1.2 M 2022-01-11T13:59:16.061 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:59:16.097 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-volume 2022-01-11T13:59:16.335 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-volume-17.0.0-10004.g17f14252.el8.noarch.r 924 kB/s | 256 kB 00:00 2022-01-11T13:59:16.336 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:16.337 INFO:teuthology.orchestra.run.smithi161.stdout:Total 918 kB/s | 256 kB 00:00 2022-01-11T13:59:16.337 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:59:16.349 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:59:16.349 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:59:16.397 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:59:16.400 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:59:16.582 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:53 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:16.607 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:59:16.908 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-volume-2:17.0.0-10004.g17f14252.el8.noarch 1/1 2022-01-11T13:59:17.269 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-volume-2:17.0.0-10004.g17f14252.el8.noarch 1/1 2022-01-11T13:59:17.469 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:17.470 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:17.470 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2022-01-11T13:59:17.470 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:17.470 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:59:17.471 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume noarch 2:17.0.0-10004.g17f14252.el8 ceph-noarch 256 k 2022-01-11T13:59:17.471 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:17.471 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:59:17.471 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:17.471 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2022-01-11T13:59:17.472 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:17.472 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 256 k 2022-01-11T13:59:17.472 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 1.2 M 2022-01-11T13:59:17.472 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:59:17.549 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-volume-2:17.0.0-10004.g17f14252.el8.noarch 1/1 2022-01-11T13:59:17.549 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:17.549 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:59:17.549 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:59:17.550 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:17.550 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:17.733 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librados-devel 2022-01-11T13:59:17.747 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-volume-17.0.0-10004.g17f14252.el8.noarch.r 926 kB/s | 256 kB 00:00 2022-01-11T13:59:17.747 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:17.748 INFO:teuthology.orchestra.run.smithi172.stdout:Total 922 kB/s | 256 kB 00:00 2022-01-11T13:59:17.748 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:59:17.759 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:59:17.759 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:59:17.777 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:59:17.780 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:59:17.958 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:59:18.225 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:55 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:18.236 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-volume-2:17.0.0-10004.g17f14252.el8.noarch 1/1 2022-01-11T13:59:18.612 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-volume-2:17.0.0-10004.g17f14252.el8.noarch 1/1 2022-01-11T13:59:18.891 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-volume-2:17.0.0-10004.g17f14252.el8.noarch 1/1 2022-01-11T13:59:18.891 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:18.891 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:59:18.891 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume-2:17.0.0-10004.g17f14252.el8.noarch 2022-01-11T13:59:18.892 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:18.892 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:19.046 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados-devel 2022-01-11T13:59:19.107 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:19.108 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:19.109 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2022-01-11T13:59:19.109 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:19.109 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:59:19.109 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel x86_64 2:17.0.0-10004.g17f14252.el8 ceph 125 k 2022-01-11T13:59:19.110 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:19.110 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:59:19.110 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:19.110 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2022-01-11T13:59:19.110 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:19.111 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 125 k 2022-01-11T13:59:19.111 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 432 k 2022-01-11T13:59:19.111 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:59:19.376 INFO:teuthology.orchestra.run.smithi161.stdout:librados-devel-17.0.0-10004.g17f14252.el8.x86_6 466 kB/s | 125 kB 00:00 2022-01-11T13:59:19.377 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:19.378 INFO:teuthology.orchestra.run.smithi161.stdout:Total 464 kB/s | 125 kB 00:00 2022-01-11T13:59:19.378 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:59:19.385 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:59:19.385 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:59:19.436 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:59:19.439 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:59:19.536 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:56 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:19.552 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:59:19.739 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librados-devel-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:19.993 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados-devel-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:20.278 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados-devel-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:20.279 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:20.280 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:59:20.280 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:59:20.280 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:20.281 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:20.443 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:20.445 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:20.446 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2022-01-11T13:59:20.446 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:20.447 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:59:20.447 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel x86_64 2:17.0.0-10004.g17f14252.el8 ceph 125 k 2022-01-11T13:59:20.448 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:20.448 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:59:20.449 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:20.449 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2022-01-11T13:59:20.449 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:20.450 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 125 k 2022-01-11T13:59:20.450 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 432 k 2022-01-11T13:59:20.451 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:59:20.452 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install libcephfs2 2022-01-11T13:59:20.737 INFO:teuthology.orchestra.run.smithi172.stdout:librados-devel-17.0.0-10004.g17f14252.el8.x86_6 425 kB/s | 125 kB 00:00 2022-01-11T13:59:20.738 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:20.739 INFO:teuthology.orchestra.run.smithi172.stdout:Total 423 kB/s | 125 kB 00:00 2022-01-11T13:59:20.739 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:59:20.747 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:59:20.747 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:59:20.796 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:59:20.799 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:59:20.908 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:59:20.942 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:57 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:21.143 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados-devel-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:21.385 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados-devel-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:21.664 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados-devel-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:21.664 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:21.664 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:59:21.665 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:59:21.665 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:21.665 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:21.824 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs2 2022-01-11T13:59:21.919 INFO:teuthology.orchestra.run.smithi161.stdout:Package libcephfs2-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:59:22.155 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:22.156 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2022-01-11T13:59:22.156 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:22.213 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install libcephfs-devel 2022-01-11T13:59:22.311 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:59 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:22.692 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:01:59 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:22.997 INFO:teuthology.orchestra.run.smithi172.stdout:Package libcephfs2-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:59:23.233 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:23.234 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2022-01-11T13:59:23.234 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:23.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs-devel 2022-01-11T13:59:23.575 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:23.577 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:23.577 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2022-01-11T13:59:23.577 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:23.578 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:59:23.578 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel x86_64 2:17.0.0-10004.g17f14252.el8 ceph 25 k 2022-01-11T13:59:23.578 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:23.578 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:59:23.579 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:23.579 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2022-01-11T13:59:23.579 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:23.579 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 25 k 2022-01-11T13:59:23.580 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 106 k 2022-01-11T13:59:23.580 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:59:23.773 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:00 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:23.774 INFO:teuthology.orchestra.run.smithi161.stdout:libcephfs-devel-17.0.0-10004.g17f14252.el8.x86_ 124 kB/s | 25 kB 00:00 2022-01-11T13:59:23.775 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:23.775 INFO:teuthology.orchestra.run.smithi161.stdout:Total 124 kB/s | 25 kB 00:00 2022-01-11T13:59:23.776 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:59:23.779 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:59:23.780 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:59:23.781 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:59:23.784 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:59:23.834 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:59:24.030 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs-devel-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:24.239 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:24.484 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs-devel-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:24.485 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:24.485 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:59:24.485 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:59:24.486 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:24.486 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:24.552 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librados2 2022-01-11T13:59:24.679 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:24.680 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2022-01-11T13:59:24.681 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:24.681 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:59:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel x86_64 2:17.0.0-10004.g17f14252.el8 ceph 25 k 2022-01-11T13:59:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:24.681 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:59:24.682 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:24.682 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2022-01-11T13:59:24.682 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:24.682 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 25 k 2022-01-11T13:59:24.683 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 106 k 2022-01-11T13:59:24.683 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:59:25.028 INFO:teuthology.orchestra.run.smithi172.stdout:libcephfs-devel-17.0.0-10004.g17f14252.el8.x86_ 71 kB/s | 25 kB 00:00 2022-01-11T13:59:25.029 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:25.029 INFO:teuthology.orchestra.run.smithi172.stdout:Total 70 kB/s | 25 kB 00:00 2022-01-11T13:59:25.030 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:59:25.030 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:02 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:25.034 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:59:25.034 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:59:25.041 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:59:25.042 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:59:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:59:25.300 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs-devel-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:25.518 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:25.692 INFO:teuthology.orchestra.run.smithi161.stdout:Package librados2-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:59:25.931 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:25.932 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2022-01-11T13:59:25.933 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:25.987 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librbd1 2022-01-11T13:59:26.116 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs-devel-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:26.116 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:26.116 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:59:26.116 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:59:26.117 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:26.117 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:26.470 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:03 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:27.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados2 2022-01-11T13:59:27.278 INFO:teuthology.orchestra.run.smithi161.stdout:Package librbd1-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:59:27.520 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:27.521 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2022-01-11T13:59:27.521 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:27.626 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:04 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:27.912 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rados 2022-01-11T13:59:28.279 INFO:teuthology.orchestra.run.smithi172.stdout:Package librados2-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:59:28.396 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:05 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:28.525 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:28.869 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2022-01-11T13:59:28.869 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:28.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librbd1 2022-01-11T13:59:29.050 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-rados-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:59:29.300 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:29.301 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2022-01-11T13:59:29.302 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:29.350 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:06 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:29.364 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rgw 2022-01-11T13:59:29.845 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:06 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:30.038 INFO:teuthology.orchestra.run.smithi172.stdout:Package librbd1-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:59:30.284 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:30.285 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2022-01-11T13:59:30.285 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:30.347 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rados 2022-01-11T13:59:30.475 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-rgw-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:59:30.714 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:30.716 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2022-01-11T13:59:30.716 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:30.769 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-cephfs 2022-01-11T13:59:30.828 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:07 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:31.249 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:08 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:31.496 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rados-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:59:31.738 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:31.739 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2022-01-11T13:59:31.739 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:31.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rgw 2022-01-11T13:59:31.925 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-cephfs-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:59:32.171 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:32.172 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2022-01-11T13:59:32.172 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:32.239 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rbd 2022-01-11T13:59:32.277 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:09 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:32.721 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:09 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:32.962 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rgw-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:59:33.209 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:33.211 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2022-01-11T13:59:33.211 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:33.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-cephfs 2022-01-11T13:59:33.358 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-rbd-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:59:33.598 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:33.599 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2022-01-11T13:59:33.600 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:33.680 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-fuse 2022-01-11T13:59:33.751 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:10 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:34.159 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:11 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:34.395 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-cephfs-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:59:34.634 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:34.635 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2022-01-11T13:59:34.635 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:34.693 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rbd 2022-01-11T13:59:35.057 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:35.058 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:35.058 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2022-01-11T13:59:35.059 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:35.059 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:59:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse x86_64 2:17.0.0-10004.g17f14252.el8 ceph 93 k 2022-01-11T13:59:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:35.060 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:59:35.060 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:35.060 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2022-01-11T13:59:35.060 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:35.061 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 93 k 2022-01-11T13:59:35.061 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 281 k 2022-01-11T13:59:35.061 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:59:35.175 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:12 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:35.307 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-fuse-17.0.0-10004.g17f14252.el8.x86_64.rpm 377 kB/s | 93 kB 00:00 2022-01-11T13:59:35.308 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:35.308 INFO:teuthology.orchestra.run.smithi161.stdout:Total 375 kB/s | 93 kB 00:00 2022-01-11T13:59:35.308 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:59:35.319 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:59:35.320 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:59:35.369 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:59:35.372 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:59:35.501 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:59:35.728 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:35.828 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rbd-2:17.0.0-10004.g17f14252.el8.x86_64 is already installed. 2022-01-11T13:59:35.961 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:36.068 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:36.069 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2022-01-11T13:59:36.069 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:36.126 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-fuse 2022-01-11T13:59:36.269 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:36.270 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:36.270 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:59:36.270 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:59:36.271 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:36.271 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:36.416 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-mirror 2022-01-11T13:59:36.607 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:13 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:36.903 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:13 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:37.516 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:37.517 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:37.517 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2022-01-11T13:59:37.518 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:37.518 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:59:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse x86_64 2:17.0.0-10004.g17f14252.el8 ceph 93 k 2022-01-11T13:59:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:37.518 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:59:37.519 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:37.519 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2022-01-11T13:59:37.519 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:37.519 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 93 k 2022-01-11T13:59:37.520 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 281 k 2022-01-11T13:59:37.520 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:59:37.730 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-fuse-17.0.0-10004.g17f14252.el8.x86_64.rpm 440 kB/s | 93 kB 00:00 2022-01-11T13:59:37.731 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:37.731 INFO:teuthology.orchestra.run.smithi172.stdout:Total 438 kB/s | 93 kB 00:00 2022-01-11T13:59:37.731 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:59:37.743 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:59:37.743 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:59:37.777 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:37.777 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:37.777 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2022-01-11T13:59:37.778 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:37.778 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:59:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror x86_64 2:17.0.0-10004.g17f14252.el8 ceph 3.8 M 2022-01-11T13:59:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:37.779 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:59:37.779 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:37.779 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2022-01-11T13:59:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:37.780 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 3.8 M 2022-01-11T13:59:37.780 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 19 M 2022-01-11T13:59:37.781 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:59:37.791 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:59:37.794 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:59:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:59:38.297 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-mirror-17.0.0-10004.g17f14252.el8.x86_64.rp 7.3 MB/s | 3.8 MB 00:00 2022-01-11T13:59:38.298 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:38.299 INFO:teuthology.orchestra.run.smithi161.stdout:Total 7.3 MB/s | 3.8 MB 00:00 2022-01-11T13:59:38.299 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:59:38.312 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:59:38.313 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:59:38.376 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:59:38.379 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:59:38.691 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:59:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:39.055 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-mirror-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:39.247 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:39.248 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:59:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:59:39.248 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:39.248 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:39.423 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-mirror 2022-01-11T13:59:39.428 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-mirror-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:39.912 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:16 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-mirror-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:40.134 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:59:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:59:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:40.135 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:40.282 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-nbd 2022-01-11T13:59:40.769 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:02:17 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:40.797 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:40.798 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:40.798 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2022-01-11T13:59:40.798 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:40.799 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:59:40.799 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror x86_64 2:17.0.0-10004.g17f14252.el8 ceph 3.8 M 2022-01-11T13:59:40.799 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:40.799 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:59:40.800 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:40.800 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2022-01-11T13:59:40.800 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:40.800 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 3.8 M 2022-01-11T13:59:40.800 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 19 M 2022-01-11T13:59:40.801 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:59:41.331 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-mirror-17.0.0-10004.g17f14252.el8.x86_64.rp 7.1 MB/s | 3.8 MB 00:00 2022-01-11T13:59:41.332 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:41.332 INFO:teuthology.orchestra.run.smithi172.stdout:Total 7.1 MB/s | 3.8 MB 00:00 2022-01-11T13:59:41.333 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:59:41.347 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:59:41.347 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:59:41.412 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:59:41.415 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:59:41.642 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-01-11T13:59:41.643 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:41.644 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2022-01-11T13:59:41.644 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:41.644 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-01-11T13:59:41.644 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd x86_64 2:17.0.0-10004.g17f14252.el8 ceph 174 k 2022-01-11T13:59:41.645 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:41.645 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-01-11T13:59:41.645 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-01-11T13:59:41.645 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2022-01-11T13:59:41.645 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:41.646 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 174 k 2022-01-11T13:59:41.646 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 568 k 2022-01-11T13:59:41.646 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-01-11T13:59:41.891 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:59:42.011 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-nbd-17.0.0-10004.g17f14252.el8.x86_64.rpm 475 kB/s | 174 kB 00:00 2022-01-11T13:59:42.012 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:42.012 INFO:teuthology.orchestra.run.smithi161.stdout:Total 474 kB/s | 174 kB 00:00 2022-01-11T13:59:42.013 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-01-11T13:59:42.025 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-01-11T13:59:42.026 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-01-11T13:59:42.074 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-mirror-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:42.076 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-01-11T13:59:42.079 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-01-11T13:59:42.218 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-01-11T13:59:42.444 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-nbd-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-mirror-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:42.681 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-nbd-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:42.782 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-mirror-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:42.782 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:42.783 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:59:42.783 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:59:42.783 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:42.783 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:42.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-nbd 2022-01-11T13:59:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-nbd-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:42.974 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-01-11T13:59:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:59:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:42.975 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-01-11T13:59:43.145 DEBUG:teuthology.parallel:result is None 2022-01-11T13:59:43.393 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:20 ago on Tue 11 Jan 2022 01:57:23 PM UTC. 2022-01-11T13:59:44.316 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2022-01-11T13:59:44.317 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:44.317 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2022-01-11T13:59:44.318 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:44.318 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2022-01-11T13:59:44.318 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd x86_64 2:17.0.0-10004.g17f14252.el8 ceph 174 k 2022-01-11T13:59:44.318 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:44.319 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2022-01-11T13:59:44.319 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2022-01-11T13:59:44.319 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2022-01-11T13:59:44.319 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:44.319 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 174 k 2022-01-11T13:59:44.320 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 568 k 2022-01-11T13:59:44.320 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2022-01-11T13:59:44.569 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-nbd-17.0.0-10004.g17f14252.el8.x86_64.rpm 695 kB/s | 174 kB 00:00 2022-01-11T13:59:44.570 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2022-01-11T13:59:44.570 INFO:teuthology.orchestra.run.smithi172.stdout:Total 692 kB/s | 174 kB 00:00 2022-01-11T13:59:44.570 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2022-01-11T13:59:44.583 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2022-01-11T13:59:44.583 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2022-01-11T13:59:44.634 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2022-01-11T13:59:44.637 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2022-01-11T13:59:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2022-01-11T13:59:44.972 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-nbd-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:45.233 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-nbd-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:45.555 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-nbd-2:17.0.0-10004.g17f14252.el8.x86_64 1/1 2022-01-11T13:59:45.555 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:45.555 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2022-01-11T13:59:45.555 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd-2:17.0.0-10004.g17f14252.el8.x86_64 2022-01-11T13:59:45.556 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:45.556 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2022-01-11T13:59:45.726 DEBUG:teuthology.parallel:result is None 2022-01-11T13:59:45.727 DEBUG:teuthology.orchestra.run.smithi161:> sudo lsb_release -is 2022-01-11T13:59:45.768 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS 2022-01-11T13:59:45.769 DEBUG:teuthology.misc:System to be installed: CentOS 2022-01-11T13:59:45.770 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=17f142526478a11e8f62d3933be552d546c4fd19 2022-01-11T13:59:45.986 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-01-11T13:59:46.009 INFO:teuthology.orchestra.run.smithi161.stdout:17.0.0-10004.g17f14252.el8 2022-01-11T13:59:46.011 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-10004.g17f14252.el8 2022-01-11T13:59:46.011 INFO:teuthology.task.install:The correct ceph version 17.0.0-10004.g17f14252 is installed. 2022-01-11T13:59:46.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo lsb_release -is 2022-01-11T13:59:46.046 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS 2022-01-11T13:59:46.047 DEBUG:teuthology.misc:System to be installed: CentOS 2022-01-11T13:59:46.048 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=17f142526478a11e8f62d3933be552d546c4fd19 2022-01-11T13:59:46.274 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-01-11T13:59:46.295 INFO:teuthology.orchestra.run.smithi172.stdout:17.0.0-10004.g17f14252.el8 2022-01-11T13:59:46.296 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-10004.g17f14252.el8 2022-01-11T13:59:46.296 INFO:teuthology.task.install:The correct ceph version 17.0.0-10004.g17f14252 is installed. 2022-01-11T13:59:46.297 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-01-11T13:59:46.297 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:46.298 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-01-11T13:59:46.326 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T13:59:46.327 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-01-11T13:59:46.360 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-01-11T13:59:46.361 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:46.361 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/daemon-helper 2022-01-11T13:59:46.393 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-01-11T13:59:46.460 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T13:59:46.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2022-01-11T13:59:46.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-01-11T13:59:46.549 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-01-11T13:59:46.549 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:46.549 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/adjust-ulimits 2022-01-11T13:59:46.578 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-01-11T13:59:46.644 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T13:59:46.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2022-01-11T13:59:46.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-01-11T13:59:46.733 INFO:teuthology.run_tasks:Running task ceph... 2022-01-11T13:59:46.795 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-01-11T13:59:46.796 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 777 /var/log/ceph 2022-01-11T13:59:46.798 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2022-01-11T13:59:46.822 INFO:tasks.ceph:Disabling ceph logrotate... 2022-01-11T13:59:46.823 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/logrotate.d/ceph 2022-01-11T13:59:46.866 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2022-01-11T13:59:46.891 INFO:tasks.ceph:Creating extra log directories... 2022-01-11T13:59:46.892 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-01-11T13:59:46.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-01-11T13:59:46.968 INFO:tasks.ceph:Creating ceph cluster c1... 2022-01-11T13:59:46.968 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-01-11T13:59:46.969 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dang-2022-01-11_08:37:46-rgw-wip-dang-testing-distro-basic-smithi/6608060', 'branch': 'wip-dang-testing', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '6608060', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'dang-2022-01-11_08:37:46-rgw-wip-dang-testing-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.3', 'overrides': {'admin_socket': {'branch': 'wip-dang-testing'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '17f142526478a11e8f62d3933be552d546c4fd19', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '17f142526478a11e8f62d3933be552d546c4fd19'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-dang-testing', 'sha1': '17f142526478a11e8f62d3933be552d546c4fd19'}}, 'owner': 'scheduled_dang@sidious.arb.redhat.com', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '17f142526478a11e8f62d3933be552d546c4fd19', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-dang-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_17f142526478a11e8f62d3933be552d546c4fd19/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '17f142526478a11e8f62d3933be552d546c4fd19', 'targets': {'smithi161.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+BHf5FlXf0gP7b+a0ejm3R2ApbRE9mcCiTGRp/GWnfGX07cnqSVzw94iCpFRzX6J3cMV8d+NLP9mYK1TQLZTixMkl+OGh+1b6hHxy/MgCPBVK3z8hNjD3cVknHoa3bIs3sYw54hPJV/toQO705SfrzsHeMi4x5339qeAxOR2bnU3KjUI4UJ046tIg5q+DfWZNm7YTOna3Ho0amBKCk5qydCf6LG1ZJP5vYjhGadjW1X9h0r+OhsMaVYf4p6CI7SEDEAy+ZDStL8pwDEes8SAXagWUe39jnXq7hLdg1R9O9Z7Vqf4v2NpTv1sqP6ViqY3fq7rGyYxdZEfw0K/sPufQLlGoW5Vt/Uevi+iu1z5p0pObvUsJwDvQ3QiEzh9bmTQ3BY3nxssCWYRzV4G5iYoUGfUQ3df0bI2tLxQ6Ox38sZUHtFCe5wWkFPh8woIlGhrp9ujJkciUSt+9QtHcfq9O+fw4v0LMBftbSlJK+nYwxGpx8ofcfKk65QS/sFiGDo0=', 'smithi172.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJy9sSSPAjBB6dmli0gWyyA/2N0akOcMO4MO8j5C90Y0c+2I7/6VsrqpueZds/gd9DvL83u49CBfO8h3Zjl5a9HRntGaDGZAqj8DM+3y/1/G2LzgJ2j5WUxIHcdJlRfNWxmGp3iQO0+Dnx0NtQY/JE94GEoLgI+3C8CGaxPnLDQWuNHK+1OMEHUGOppA41xNRzPm7NwEWKEtAh8sa9Fx3VaCXHSqvdxjY1KFg5xSvKkpCF5Wkt60pb8w0GHwM4HqtbbBOwuDIbwQa3cH4ZnriHqypTyVg48xmBOjXjLeClyxete2ExIxSgjlKigQYdU9PKXGO7D+wFunduTokQSA5X2IRH30cy3UvFZyYZpoPHymZCPg+ZJMHic12xNdA36MiSxFOdE/y3ZkQ7Apd+Sz4D6W7+RdBXBPHu67bWf6GCh4WJdCXs1WKAaIimt0SHeNCmYl+jgWy04IgGkm6fd4lGD0VMs+Ub/qj/P8pmyGEzjmizhoBpXpl//Od+07hBfh8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '17f142526478a11e8f62d3933be552d546c4fd19', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '00e09f05e9fb2ab7c7f497f5e446613531743ae2', 'timestamp': '2022-01-11_08:37:46', 'tube': 'smithi', 'user': 'dang', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.20261'} 2022-01-11T13:59:46.970 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2022-01-11T13:59:47.000 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2022-01-11T13:59:47.015 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/run/ceph 2022-01-11T13:59:47.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2022-01-11T13:59:47.082 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:47.082 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2022-01-11T13:59:47.133 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-01-11T13:59:47.134 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-01-11T13:59:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-01-11T13:59:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-11T13:59:47.188 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 6h/6d Inode: 115150 Links: 1 2022-01-11T13:59:47.188 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-11T13:59:47.188 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2022-01-11T13:59:47.189 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2022-01-11 13:59:39.223550655 +0000 2022-01-11T13:59:47.189 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2022-01-11 13:56:23.291959629 +0000 2022-01-11T13:59:47.189 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2022-01-11 13:56:23.291959629 +0000 2022-01-11T13:59:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2022-01-11T13:59:47.197 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2022-01-11T13:59:47.197 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2022-01-11T13:59:47.198 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000255091 s, 2.0 MB/s 2022-01-11T13:59:47.200 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-01-11T13:59:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-01-11T13:59:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-11T13:59:47.256 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 6h/6d Inode: 110318 Links: 1 2022-01-11T13:59:47.257 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-11T13:59:47.257 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2022-01-11T13:59:47.257 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2022-01-11 13:59:39.223550655 +0000 2022-01-11T13:59:47.257 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2022-01-11 13:56:23.640958988 +0000 2022-01-11T13:59:47.257 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2022-01-11 13:56:23.640958988 +0000 2022-01-11T13:59:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2022-01-11T13:59:47.265 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2022-01-11T13:59:47.266 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2022-01-11T13:59:47.267 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000274247 s, 1.9 MB/s 2022-01-11T13:59:47.268 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-01-11T13:59:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-01-11T13:59:47.324 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-11T13:59:47.324 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 6h/6d Inode: 115742 Links: 1 2022-01-11T13:59:47.324 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-11T13:59:47.324 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2022-01-11T13:59:47.325 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2022-01-11 13:59:39.223550655 +0000 2022-01-11T13:59:47.325 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2022-01-11 13:56:23.974958374 +0000 2022-01-11T13:59:47.325 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2022-01-11 13:56:23.974958374 +0000 2022-01-11T13:59:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2022-01-11T13:59:47.333 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2022-01-11T13:59:47.333 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2022-01-11T13:59:47.333 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000234598 s, 2.2 MB/s 2022-01-11T13:59:47.335 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-01-11T13:59:47.390 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-01-11T13:59:47.390 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-11T13:59:47.391 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 6h/6d Inode: 115295 Links: 1 2022-01-11T13:59:47.391 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-11T13:59:47.391 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2022-01-11T13:59:47.392 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2022-01-11 13:59:39.223550655 +0000 2022-01-11T13:59:47.392 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2022-01-11 13:56:24.308957761 +0000 2022-01-11T13:59:47.393 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2022-01-11 13:56:24.308957761 +0000 2022-01-11T13:59:47.393 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2022-01-11T13:59:47.399 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2022-01-11T13:59:47.400 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2022-01-11T13:59:47.400 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000195296 s, 2.6 MB/s 2022-01-11T13:59:47.402 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-01-11T13:59:47.403 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2022-01-11T13:59:47.403 INFO:tasks.ceph:Generating config... 2022-01-11T13:59:47.404 INFO:tasks.ceph:[client] debug rgw = 20 2022-01-11T13:59:47.405 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2022-01-11T13:59:47.405 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2022-01-11T13:59:47.405 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-01-11T13:59:47.406 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2022-01-11T13:59:47.406 INFO:tasks.ceph:[client] rgw data log num shards = 4 2022-01-11T13:59:47.406 INFO:tasks.ceph:[client] rgw md log max shards = 4 2022-01-11T13:59:47.407 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2022-01-11T13:59:47.407 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2022-01-11T13:59:47.407 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2022-01-11T13:59:47.408 INFO:tasks.ceph:[client] setgroup = ceph 2022-01-11T13:59:47.408 INFO:tasks.ceph:[client] setuser = ceph 2022-01-11T13:59:47.408 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-01-11T13:59:47.409 INFO:tasks.ceph:[mgr] debug ms = 1 2022-01-11T13:59:47.409 INFO:tasks.ceph:[mon] debug mon = 20 2022-01-11T13:59:47.410 INFO:tasks.ceph:[mon] debug ms = 1 2022-01-11T13:59:47.410 INFO:tasks.ceph:[mon] debug paxos = 20 2022-01-11T13:59:47.410 INFO:tasks.ceph:[osd] debug ms = 1 2022-01-11T13:59:47.411 INFO:tasks.ceph:[osd] debug osd = 20 2022-01-11T13:59:47.411 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2022-01-11T13:59:47.411 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2022-01-11T13:59:47.412 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2022-01-11T13:59:47.412 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2022-01-11T13:59:47.412 INFO:tasks.ceph:Setting up c1.mon.a... 2022-01-11T13:59:47.413 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2022-01-11T13:59:47.514 INFO:teuthology.orchestra.run.smithi161.stdout:creating /etc/ceph/c1.keyring 2022-01-11T13:59:47.516 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2022-01-11T13:59:47.555 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-01-11T13:59:47.618 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.161')] 2022-01-11T13:59:47.619 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.161'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2022-01-11T13:59:47.620 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:47.620 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-01-11T13:59:47.674 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.161 --print /home/ubuntu/cephtest/c1.monmap 2022-01-11T13:59:47.749 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T13:59:47.749 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T13:59:47.751 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2022-01-11T13:59:47.751 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: generated fsid fd8a9b68-e58b-4f3b-8917-54d7aadd438b 2022-01-11T13:59:47.751 INFO:teuthology.orchestra.run.smithi161.stdout:setting min_mon_release = octopus 2022-01-11T13:59:47.751 INFO:teuthology.orchestra.run.smithi161.stdout:epoch 0 2022-01-11T13:59:47.752 INFO:teuthology.orchestra.run.smithi161.stdout:fsid fd8a9b68-e58b-4f3b-8917-54d7aadd438b 2022-01-11T13:59:47.752 INFO:teuthology.orchestra.run.smithi161.stdout:last_changed 2022-01-11T13:59:47.752766+0000 2022-01-11T13:59:47.752 INFO:teuthology.orchestra.run.smithi161.stdout:created 2022-01-11T13:59:47.752766+0000 2022-01-11T13:59:47.752 INFO:teuthology.orchestra.run.smithi161.stdout:min_mon_release 15 (octopus) 2022-01-11T13:59:47.753 INFO:teuthology.orchestra.run.smithi161.stdout:election_strategy: 1 2022-01-11T13:59:47.753 INFO:teuthology.orchestra.run.smithi161.stdout:0: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.a 2022-01-11T13:59:47.753 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2022-01-11T13:59:47.754 DEBUG:teuthology.orchestra.run.smithi161:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-01-11T13:59:47.806 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID fd8a9b68-e58b-4f3b-8917-54d7aadd438b... 2022-01-11T13:59:47.807 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2022-01-11T13:59:47.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2022-01-11T13:59:47.887 INFO:teuthology.orchestra.run.smithi161.stdout:[global] 2022-01-11T13:59:47.888 INFO:teuthology.orchestra.run.smithi161.stdout: chdir = "" 2022-01-11T13:59:47.888 INFO:teuthology.orchestra.run.smithi161.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-01-11T13:59:47.888 INFO:teuthology.orchestra.run.smithi161.stdout: auth supported = cephx 2022-01-11T13:59:47.888 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.889 INFO:teuthology.orchestra.run.smithi161.stdout: filestore xattr use omap = true 2022-01-11T13:59:47.889 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.889 INFO:teuthology.orchestra.run.smithi161.stdout: mon clock drift allowed = 1.000 2022-01-11T13:59:47.889 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.890 INFO:teuthology.orchestra.run.smithi161.stdout: osd crush chooseleaf type = 0 2022-01-11T13:59:47.890 INFO:teuthology.orchestra.run.smithi161.stdout: auth debug = true 2022-01-11T13:59:47.890 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.890 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on old message = true 2022-01-11T13:59:47.891 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on bug = true 2022-01-11T13:59:47.891 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.891 INFO:teuthology.orchestra.run.smithi161.stdout: mon max pg per osd = 10000 # >= luminous 2022-01-11T13:59:47.892 INFO:teuthology.orchestra.run.smithi161.stdout: mon pg warn max object skew = 0 2022-01-11T13:59:47.892 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.892 INFO:teuthology.orchestra.run.smithi161.stdout: # disable pg_autoscaler by default for new pools 2022-01-11T13:59:47.893 INFO:teuthology.orchestra.run.smithi161.stdout: osd_pool_default_pg_autoscale_mode = off 2022-01-11T13:59:47.893 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.893 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default size = 2 2022-01-11T13:59:47.893 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.894 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow primary affinity = true 2022-01-11T13:59:47.894 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow pg remap = true 2022-01-11T13:59:47.894 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on legacy crush tunables = false 2022-01-11T13:59:47.894 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on crush straw calc version zero = false 2022-01-11T13:59:47.895 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on no sortbitwise = false 2022-01-11T13:59:47.895 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on osd down out interval zero = false 2022-01-11T13:59:47.895 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on too few osds = false 2022-01-11T13:59:47.896 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-01-11T13:59:47.896 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_no_redundancy = false 2022-01-11T13:59:47.896 INFO:teuthology.orchestra.run.smithi161.stdout: mon_allow_pool_size_one = true 2022-01-11T13:59:47.896 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.897 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-01-11T13:59:47.897 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.897 INFO:teuthology.orchestra.run.smithi161.stdout: osd default data pool replay window = 5 2022-01-11T13:59:47.897 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.897 INFO:teuthology.orchestra.run.smithi161.stdout: mon allow pool delete = true 2022-01-11T13:59:47.898 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.898 INFO:teuthology.orchestra.run.smithi161.stdout: mon cluster log file level = debug 2022-01-11T13:59:47.898 INFO:teuthology.orchestra.run.smithi161.stdout: debug asserts on shutdown = true 2022-01-11T13:59:47.899 INFO:teuthology.orchestra.run.smithi161.stdout: mon health detail to clog = false 2022-01-11T13:59:47.899 INFO:teuthology.orchestra.run.smithi161.stdout: mon host = 172.21.15.161 2022-01-11T13:59:47.899 INFO:teuthology.orchestra.run.smithi161.stdout: fsid = fd8a9b68-e58b-4f3b-8917-54d7aadd438b 2022-01-11T13:59:47.899 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.900 INFO:teuthology.orchestra.run.smithi161.stdout:[osd] 2022-01-11T13:59:47.900 INFO:teuthology.orchestra.run.smithi161.stdout: osd journal size = 100 2022-01-11T13:59:47.900 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.900 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub load threshold = 5.0 2022-01-11T13:59:47.901 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub max interval = 600 2022-01-11T13:59:47.901 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.901 INFO:teuthology.orchestra.run.smithi161.stdout: osd recover clone overlap = true 2022-01-11T13:59:47.901 INFO:teuthology.orchestra.run.smithi161.stdout: osd recovery max chunk = 1048576 2022-01-11T13:59:47.901 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.902 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug shutdown = true 2022-01-11T13:59:47.902 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug op order = true 2022-01-11T13:59:47.902 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug verify stray on activate = true 2022-01-11T13:59:47.902 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.903 INFO:teuthology.orchestra.run.smithi161.stdout: osd open classes on start = true 2022-01-11T13:59:47.903 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug pg log writeout = true 2022-01-11T13:59:47.903 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.903 INFO:teuthology.orchestra.run.smithi161.stdout: osd deep scrub update digest min age = 30 2022-01-11T13:59:47.904 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.904 INFO:teuthology.orchestra.run.smithi161.stdout: osd map max advance = 10 2022-01-11T13:59:47.904 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.904 INFO:teuthology.orchestra.run.smithi161.stdout: journal zero on create = true 2022-01-11T13:59:47.904 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.905 INFO:teuthology.orchestra.run.smithi161.stdout: filestore ondisk finisher threads = 3 2022-01-11T13:59:47.905 INFO:teuthology.orchestra.run.smithi161.stdout: filestore apply finisher threads = 3 2022-01-11T13:59:47.905 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.905 INFO:teuthology.orchestra.run.smithi161.stdout: bdev debug aio = true 2022-01-11T13:59:47.906 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug misdirected ops = true 2022-01-11T13:59:47.906 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2022-01-11T13:59:47.906 INFO:teuthology.orchestra.run.smithi161.stdout: debug osd = 20 2022-01-11T13:59:47.907 INFO:teuthology.orchestra.run.smithi161.stdout: rocksdb delete range threshold = 0 2022-01-11T13:59:47.907 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.907 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr] 2022-01-11T13:59:47.907 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2022-01-11T13:59:47.908 INFO:teuthology.orchestra.run.smithi161.stdout: debug mgr = 20 2022-01-11T13:59:47.908 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2022-01-11T13:59:47.908 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2022-01-11T13:59:47.908 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2022-01-11T13:59:47.908 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2022-01-11T13:59:47.909 INFO:teuthology.orchestra.run.smithi161.stdout: mgr/telemetry/nag = false 2022-01-11T13:59:47.909 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.909 INFO:teuthology.orchestra.run.smithi161.stdout:[mon] 2022-01-11T13:59:47.909 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2022-01-11T13:59:47.910 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2022-01-11T13:59:47.910 INFO:teuthology.orchestra.run.smithi161.stdout: debug paxos = 20 2022-01-11T13:59:47.910 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2022-01-11T13:59:47.910 INFO:teuthology.orchestra.run.smithi161.stdout: mon data avail warn = 5 2022-01-11T13:59:47.910 INFO:teuthology.orchestra.run.smithi161.stdout: mon mgr mkfs grace = 240 2022-01-11T13:59:47.911 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2022-01-11T13:59:47.911 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd reporter subtree level = osd 2022-01-11T13:59:47.911 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd prime pg temp = true 2022-01-11T13:59:47.911 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2022-01-11T13:59:47.912 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.912 INFO:teuthology.orchestra.run.smithi161.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-01-11T13:59:47.912 INFO:teuthology.orchestra.run.smithi161.stdout: auth mon ticket ttl = 660 # 11m 2022-01-11T13:59:47.912 INFO:teuthology.orchestra.run.smithi161.stdout: auth service ticket ttl = 240 # 4m 2022-01-11T13:59:47.913 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.913 INFO:teuthology.orchestra.run.smithi161.stdout: # don't complain about insecure global_id in the test suite 2022-01-11T13:59:47.913 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-01-11T13:59:47.913 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-01-11T13:59:47.913 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.914 INFO:teuthology.orchestra.run.smithi161.stdout: # 1m isn't quite enough 2022-01-11T13:59:47.914 INFO:teuthology.orchestra.run.smithi161.stdout: mon_down_mkfs_grace = 2m 2022-01-11T13:59:47.914 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:47.914 INFO:teuthology.orchestra.run.smithi161.stdout:[client] 2022-01-11T13:59:47.915 INFO:teuthology.orchestra.run.smithi161.stdout: rgw cache enabled = true 2022-01-11T13:59:47.915 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable ops log = true 2022-01-11T13:59:47.915 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable usage log = true 2022-01-11T13:59:47.915 INFO:teuthology.orchestra.run.smithi161.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-01-11T13:59:47.915 INFO:teuthology.orchestra.run.smithi161.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-01-11T13:59:47.916 INFO:teuthology.orchestra.run.smithi161.stdout: debug rgw = 20 2022-01-11T13:59:47.916 INFO:teuthology.orchestra.run.smithi161.stdout: rgw crypt require ssl = False 2022-01-11T13:59:47.916 INFO:teuthology.orchestra.run.smithi161.stdout: rgw crypt s3 kms backend = testing 2022-01-11T13:59:47.917 INFO:teuthology.orchestra.run.smithi161.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-01-11T13:59:47.917 INFO:teuthology.orchestra.run.smithi161.stdout: rgw curl low speed time = 300 2022-01-11T13:59:47.917 INFO:teuthology.orchestra.run.smithi161.stdout: rgw data log num shards = 4 2022-01-11T13:59:47.917 INFO:teuthology.orchestra.run.smithi161.stdout: rgw md log max shards = 4 2022-01-11T13:59:47.918 INFO:teuthology.orchestra.run.smithi161.stdout: rgw sync log trim interval = 0 2022-01-11T13:59:47.918 INFO:teuthology.orchestra.run.smithi161.stdout: rgw sync meta inject err probability = 0.1 2022-01-11T13:59:47.918 INFO:teuthology.orchestra.run.smithi161.stdout: rgw sync obj etag verify = True 2022-01-11T13:59:47.918 INFO:teuthology.orchestra.run.smithi161.stdout: setgroup = ceph 2022-01-11T13:59:47.919 INFO:teuthology.orchestra.run.smithi161.stdout: setuser = ceph 2022-01-11T13:59:47.919 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.a] 2022-01-11T13:59:47.919 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.0] 2022-01-11T13:59:47.919 INFO:teuthology.orchestra.run.smithi161.stdout: osd_max_omap_entries_per_request = 10 2022-01-11T13:59:47.919 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.1] 2022-01-11T13:59:47.920 INFO:teuthology.orchestra.run.smithi161.stdout: osd_max_omap_entries_per_request = 1000 2022-01-11T13:59:47.920 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.2] 2022-01-11T13:59:47.920 INFO:teuthology.orchestra.run.smithi161.stdout: osd_max_omap_entries_per_request = 10000 2022-01-11T13:59:47.922 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2022-01-11T13:59:47.922 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2022-01-11T13:59:47.922 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-01-11T13:59:47.923 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2022-01-11T13:59:47.923 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.923 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2022-01-11T13:59:47.923 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.924 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2022-01-11T13:59:47.924 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.924 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2022-01-11T13:59:47.925 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2022-01-11T13:59:47.925 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.925 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2022-01-11T13:59:47.926 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2022-01-11T13:59:47.926 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.926 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2022-01-11T13:59:47.927 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2022-01-11T13:59:47.927 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.927 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2022-01-11T13:59:47.928 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2022-01-11T13:59:47.928 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.928 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2022-01-11T13:59:47.928 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.928 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2022-01-11T13:59:47.929 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2022-01-11T13:59:47.929 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2022-01-11T13:59:47.929 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2022-01-11T13:59:47.929 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2022-01-11T13:59:47.930 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2022-01-11T13:59:47.930 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2022-01-11T13:59:47.930 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-01-11T13:59:47.931 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2022-01-11T13:59:47.931 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2022-01-11T13:59:47.931 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.931 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-01-11T13:59:47.931 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.932 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2022-01-11T13:59:47.932 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.932 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2022-01-11T13:59:47.932 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.933 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2022-01-11T13:59:47.933 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2022-01-11T13:59:47.933 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2022-01-11T13:59:47.933 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = 172.21.15.161 2022-01-11T13:59:47.934 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = fd8a9b68-e58b-4f3b-8917-54d7aadd438b 2022-01-11T13:59:47.934 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.934 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2022-01-11T13:59:47.934 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2022-01-11T13:59:47.935 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.935 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2022-01-11T13:59:47.935 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2022-01-11T13:59:47.935 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.936 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2022-01-11T13:59:47.936 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2022-01-11T13:59:47.936 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.937 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2022-01-11T13:59:47.937 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2022-01-11T13:59:47.937 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2022-01-11T13:59:47.938 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.938 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2022-01-11T13:59:47.938 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2022-01-11T13:59:47.938 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.939 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2022-01-11T13:59:47.939 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.939 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2022-01-11T13:59:47.939 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.940 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2022-01-11T13:59:47.940 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.940 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2022-01-11T13:59:47.940 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2022-01-11T13:59:47.940 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.941 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2022-01-11T13:59:47.941 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2022-01-11T13:59:47.941 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2022-01-11T13:59:47.941 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2022-01-11T13:59:47.942 INFO:teuthology.orchestra.run.smithi172.stdout: rocksdb delete range threshold = 0 2022-01-11T13:59:47.942 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.942 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2022-01-11T13:59:47.942 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2022-01-11T13:59:47.942 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2022-01-11T13:59:47.943 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2022-01-11T13:59:47.943 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2022-01-11T13:59:47.943 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2022-01-11T13:59:47.943 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2022-01-11T13:59:47.944 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2022-01-11T13:59:47.944 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.944 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2022-01-11T13:59:47.944 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2022-01-11T13:59:47.944 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2022-01-11T13:59:47.945 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2022-01-11T13:59:47.945 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2022-01-11T13:59:47.945 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2022-01-11T13:59:47.945 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2022-01-11T13:59:47.946 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2022-01-11T13:59:47.946 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2022-01-11T13:59:47.946 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2022-01-11T13:59:47.946 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2022-01-11T13:59:47.946 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.947 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-01-11T13:59:47.947 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2022-01-11T13:59:47.947 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2022-01-11T13:59:47.948 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.948 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2022-01-11T13:59:47.948 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-01-11T13:59:47.948 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-01-11T13:59:47.949 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.949 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2022-01-11T13:59:47.949 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2022-01-11T13:59:47.949 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T13:59:47.949 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2022-01-11T13:59:47.950 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2022-01-11T13:59:47.950 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2022-01-11T13:59:47.950 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2022-01-11T13:59:47.950 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-01-11T13:59:47.951 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-01-11T13:59:47.951 INFO:teuthology.orchestra.run.smithi172.stdout: debug rgw = 20 2022-01-11T13:59:47.951 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt require ssl = False 2022-01-11T13:59:47.951 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt s3 kms backend = testing 2022-01-11T13:59:47.951 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-01-11T13:59:47.952 INFO:teuthology.orchestra.run.smithi172.stdout: rgw curl low speed time = 300 2022-01-11T13:59:47.952 INFO:teuthology.orchestra.run.smithi172.stdout: rgw data log num shards = 4 2022-01-11T13:59:47.952 INFO:teuthology.orchestra.run.smithi172.stdout: rgw md log max shards = 4 2022-01-11T13:59:47.952 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync log trim interval = 0 2022-01-11T13:59:47.953 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync meta inject err probability = 0.1 2022-01-11T13:59:47.953 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync obj etag verify = True 2022-01-11T13:59:47.953 INFO:teuthology.orchestra.run.smithi172.stdout: setgroup = ceph 2022-01-11T13:59:47.953 INFO:teuthology.orchestra.run.smithi172.stdout: setuser = ceph 2022-01-11T13:59:47.953 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2022-01-11T13:59:47.954 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2022-01-11T13:59:47.954 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 10 2022-01-11T13:59:47.954 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2022-01-11T13:59:47.954 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 1000 2022-01-11T13:59:47.955 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2022-01-11T13:59:47.955 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 10000 2022-01-11T13:59:47.955 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2022-01-11T13:59:47.956 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2022-01-11T13:59:48.012 INFO:tasks.ceph:Copying monmap to all nodes... 2022-01-11T13:59:48.012 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:48.013 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2022-01-11T13:59:48.028 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:48.028 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2022-01-11T13:59:48.082 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi161.front.sepia.ceph.com 2022-01-11T13:59:48.082 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:48.083 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/c1.keyring 2022-01-11T13:59:48.083 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-01-11T13:59:48.154 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:48.155 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/c1.monmap 2022-01-11T13:59:48.209 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2022-01-11T13:59:48.210 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T13:59:48.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/c1.keyring 2022-01-11T13:59:48.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/c1.keyring 2022-01-11T13:59:48.251 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T13:59:48.251 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/c1.monmap 2022-01-11T13:59:48.307 INFO:tasks.ceph:Setting up mon nodes... 2022-01-11T13:59:48.308 INFO:tasks.ceph:Setting up mgr nodes... 2022-01-11T13:59:48.308 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2022-01-11T13:59:48.355 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2022-01-11T13:59:48.357 INFO:tasks.ceph:Setting up mds nodes... 2022-01-11T13:59:48.357 INFO:tasks.ceph_client:Setting up client nodes... 2022-01-11T13:59:48.358 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2022-01-11T13:59:48.395 INFO:teuthology.orchestra.run.smithi161.stdout:creating /etc/ceph/c1.client.0.keyring 2022-01-11T13:59:48.407 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2022-01-11T13:59:48.484 INFO:teuthology.orchestra.run.smithi161.stdout:creating /etc/ceph/c1.client.1.keyring 2022-01-11T13:59:48.496 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-01-11T13:59:48.497 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2022-01-11T13:59:48.498 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2022-01-11T13:59:48.560 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-01-11T13:59:48.560 INFO:tasks.ceph:role: c1.osd.0 2022-01-11T13:59:48.560 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com 2022-01-11T13:59:48.561 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-01-11T13:59:48.631 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-01-11T13:59:48.632 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2022-01-11T13:59:48.632 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-01-11T13:59:48.633 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 2022-01-11T13:59:48.633 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-01-11T13:59:48.633 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2022-01-11T13:59:48.634 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-01-11T13:59:48.634 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-01-11T13:59:48.635 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-01-11T13:59:48.635 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-01-11T13:59:48.995 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2022-01-11T13:59:48.996 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2022-01-11T13:59:48.997 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2022-01-11T13:59:49.156 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2022-01-11T13:59:49.187 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2022-01-11T13:59:49.260 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-01-11T13:59:49.260 INFO:tasks.ceph:role: c1.osd.1 2022-01-11T13:59:49.260 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com 2022-01-11T13:59:49.261 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-01-11T13:59:49.331 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-01-11T13:59:49.332 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2022-01-11T13:59:49.332 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-01-11T13:59:49.332 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 2022-01-11T13:59:49.332 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-01-11T13:59:49.333 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2022-01-11T13:59:49.333 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-01-11T13:59:49.333 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-01-11T13:59:49.334 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-01-11T13:59:49.334 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-01-11T13:59:49.687 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2022-01-11T13:59:49.689 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2022-01-11T13:59:49.689 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2022-01-11T13:59:49.774 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2022-01-11T13:59:49.808 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2022-01-11T13:59:49.872 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2022-01-11T13:59:49.873 INFO:tasks.ceph:role: c1.osd.2 2022-01-11T13:59:49.873 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com 2022-01-11T13:59:49.874 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-01-11T13:59:49.944 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-01-11T13:59:49.944 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2022-01-11T13:59:49.945 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-01-11T13:59:49.945 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 2022-01-11T13:59:49.945 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-01-11T13:59:49.945 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2022-01-11T13:59:49.946 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-01-11T13:59:49.946 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-01-11T13:59:49.946 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-01-11T13:59:49.947 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-01-11T13:59:50.312 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2022-01-11T13:59:50.314 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2022-01-11T13:59:50.314 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2022-01-11T13:59:50.432 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2022-01-11T13:59:50.463 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2022-01-11T13:59:50.557 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:50.559+0000 7f8951274240 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2022-01-11T13:59:50.559 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:50.561+0000 7f8951274240 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2022-01-11T13:59:50.560 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:50.561+0000 7f8951274240 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-01-11T13:59:50.560 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:50.561+0000 7f8951274240 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-01-11T13:59:50.560 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:50.561+0000 7f8951274240 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2022-01-11T13:59:50.564 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:50.566+0000 7f8951274240 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2022-01-11T13:59:52.643 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2022-01-11T13:59:52.672 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2022-01-11T13:59:52.757 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:52.758+0000 7fdc2401a240 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2022-01-11T13:59:52.757 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:52.758+0000 7fdc2401a240 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2022-01-11T13:59:52.757 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:52.758+0000 7fdc2401a240 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-01-11T13:59:52.758 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:52.758+0000 7fdc2401a240 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-01-11T13:59:52.758 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:52.758+0000 7fdc2401a240 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2022-01-11T13:59:52.758 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:52.758+0000 7fdc2401a240 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2022-01-11T13:59:54.840 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2022-01-11T13:59:54.869 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2022-01-11T13:59:54.955 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:54.956+0000 7f98d5986240 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2022-01-11T13:59:54.955 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:54.956+0000 7f98d5986240 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2022-01-11T13:59:54.955 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:54.956+0000 7f98d5986240 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-01-11T13:59:54.956 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:54.956+0000 7f98d5986240 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-01-11T13:59:54.956 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:54.956+0000 7f98d5986240 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2022-01-11T13:59:54.956 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T13:59:54.957+0000 7f98d5986240 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2022-01-11T13:59:57.034 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2022-01-11T13:59:57.062 INFO:tasks.ceph:Reading keys from all nodes... 2022-01-11T13:59:57.062 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:57.063 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2022-01-11T13:59:57.127 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:57.128 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2022-01-11T13:59:57.192 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:57.192 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2022-01-11T13:59:57.256 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:57.257 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2022-01-11T13:59:57.321 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:57.322 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2022-01-11T13:59:57.376 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:57.376 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2022-01-11T13:59:57.431 INFO:tasks.ceph:Adding keys to all mons... 2022-01-11T13:59:57.431 DEBUG:teuthology.orchestra.run.smithi161:> sudo tee -a /etc/ceph/c1.keyring 2022-01-11T13:59:57.493 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr.x] 2022-01-11T13:59:57.494 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBUjd1hr11YFRAAUTJ1bTdYr5xnmvFd/rbnqw== 2022-01-11T13:59:57.494 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.0] 2022-01-11T13:59:57.495 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBWjd1hzV9rIRAAnWjmYt3Qy8xTSOLsZDEEUw== 2022-01-11T13:59:57.495 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.1] 2022-01-11T13:59:57.495 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBYjd1h58VELRAA4jUGWRp4hEvK82JxowFIVw== 2022-01-11T13:59:57.495 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.2] 2022-01-11T13:59:57.496 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBajd1heEISORAAEIT0aMBeH/57zGHTSto/bw== 2022-01-11T13:59:57.496 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2022-01-11T13:59:57.496 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBUjd1hZSK+FxAAZOguEIndgwLzUXbxBVNFKw== 2022-01-11T13:59:57.496 INFO:teuthology.orchestra.run.smithi161.stdout:[client.1] 2022-01-11T13:59:57.497 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBUjd1hegcQHRAAd5OQr4DrRm9Imwz9/zzsug== 2022-01-11T13:59:57.498 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-01-11T13:59:57.586 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-01-11T13:59:57.632 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-01-11T13:59:57.678 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-01-11T13:59:57.764 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-01-11T13:59:57.813 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-01-11T13:59:57.862 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-01-11T13:59:57.863 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2022-01-11T13:59:57.887 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2022-01-11T13:59:57.977 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2022-01-11T13:59:58.000 DEBUG:teuthology.orchestra.run.smithi161:> rm -- /home/ubuntu/cephtest/c1.monmap 2022-01-11T13:59:58.055 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2022-01-11T13:59:58.055 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2022-01-11T13:59:58.056 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2022-01-11T13:59:58.097 INFO:tasks.ceph.c1.mon.a:Started 2022-01-11T13:59:58.097 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2022-01-11T13:59:58.098 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2022-01-11T13:59:58.098 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2022-01-11T13:59:58.100 INFO:tasks.ceph.c1.mgr.x:Started 2022-01-11T13:59:58.100 INFO:tasks.ceph:Setting crush tunables to default 2022-01-11T13:59:58.101 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster c1 osd crush tunables default 2022-01-11T13:59:58.297 INFO:teuthology.orchestra.run.smithi161.stderr:adjusted tunables profile to default 2022-01-11T13:59:58.308 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2022-01-11T13:59:58.309 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:58.309 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2022-01-11T13:59:58.333 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:58.334 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2022-01-11T13:59:58.396 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T13:59:58.397 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2022-01-11T13:59:58.460 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster c1 osd new f27f63ff-7822-47ef-9fc6-46ca314c3052 0 2022-01-11T13:59:58.577 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T13:59:58.579+0000 7fca2a260000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-01-11T13:59:58.702 INFO:teuthology.orchestra.run.smithi161.stdout:0 2022-01-11T13:59:58.710 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T13:59:58.712+0000 7fca2a260000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-01-11T13:59:58.714 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster c1 osd new 17448e5a-bf40-441c-b631-c071795fe273 1 2022-01-11T13:59:58.782 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T13:59:58.784+0000 7fca2a260000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-01-11T13:59:58.917 INFO:teuthology.orchestra.run.smithi161.stdout:1 2022-01-11T13:59:58.929 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster c1 osd new 2b6eb01f-1796-4534-935e-0f194d39b3cb 2 2022-01-11T13:59:59.144 INFO:teuthology.orchestra.run.smithi161.stdout:2 2022-01-11T13:59:59.157 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2022-01-11T13:59:59.158 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2022-01-11T13:59:59.160 INFO:tasks.ceph.c1.osd.0:Started 2022-01-11T13:59:59.161 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2022-01-11T13:59:59.161 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2022-01-11T13:59:59.164 INFO:tasks.ceph.c1.osd.1:Started 2022-01-11T13:59:59.165 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2022-01-11T13:59:59.165 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2022-01-11T13:59:59.169 INFO:tasks.ceph.c1.osd.2:Started 2022-01-11T13:59:59.170 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-01-11T13:59:59.370 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T13:59:59.370 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":5,"fsid":"fd8a9b68-e58b-4f3b-8917-54d7aadd438b","created":"2022-01-11T13:59:58.214991+0000","modified":"2022-01-11T13:59:59.145902+0000","last_up_change":"0.000000","last_in_change":"2022-01-11T13:59:59.145902+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"f27f63ff-7822-47ef-9fc6-46ca314c3052","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"17448e5a-bf40-441c-b631-c071795fe273","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2b6eb01f-1796-4534-935e-0f194d39b3cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-11T13:59:59.382 INFO:tasks.ceph.ceph_manager.c1:[] 2022-01-11T13:59:59.383 INFO:tasks.ceph:Waiting for OSDs to come up 2022-01-11T13:59:59.510 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T13:59:59.512+0000 7fca2a260000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-01-11T13:59:59.776 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T13:59:59.577+0000 7fca2a260000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-01-11T13:59:59.777 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T13:59:59.650+0000 7fca2a260000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-01-11T13:59:59.777 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T13:59:59.715+0000 7fca2a260000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-01-11T13:59:59.778 INFO:tasks.ceph.c1.osd.1.smithi161.stderr:2022-01-11T13:59:59.761+0000 7f719e189240 -1 Falling back to public interface 2022-01-11T13:59:59.779 INFO:tasks.ceph.c1.osd.0.smithi161.stderr:2022-01-11T13:59:59.765+0000 7f8c5a09c240 -1 Falling back to public interface 2022-01-11T13:59:59.779 INFO:tasks.ceph.c1.osd.2.smithi161.stderr:2022-01-11T13:59:59.765+0000 7f03885b1240 -1 Falling back to public interface 2022-01-11T13:59:59.780 INFO:tasks.ceph.c1.osd.2.smithi161.stderr:2022-01-11T13:59:59.765+0000 7f03829da700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6808/0: (98) Address already in use 2022-01-11T13:59:59.847 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T13:59:59.849+0000 7fca2a260000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-01-11T13:59:59.876 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-01-11T13:59:59.934 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T13:59:59.935+0000 7fca2a260000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-01-11T14:00:00.061 INFO:teuthology.misc.health.smithi161.stdout: 2022-01-11T14:00:00.062 INFO:teuthology.misc.health.smithi161.stdout:{"epoch":5,"fsid":"fd8a9b68-e58b-4f3b-8917-54d7aadd438b","created":"2022-01-11T13:59:58.214991+0000","modified":"2022-01-11T13:59:59.145902+0000","last_up_change":"0.000000","last_in_change":"2022-01-11T13:59:59.145902+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"f27f63ff-7822-47ef-9fc6-46ca314c3052","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"17448e5a-bf40-441c-b631-c071795fe273","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2b6eb01f-1796-4534-935e-0f194d39b3cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-11T14:00:00.072 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-01-11T14:00:00.266 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:00.268+0000 7fca2a260000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-01-11T14:00:00.361 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:00.362+0000 7fca2a260000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-01-11T14:00:00.434 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:00.436+0000 7fca2a260000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-01-11T14:00:00.502 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:00.504+0000 7fca2a260000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-01-11T14:00:00.771 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:00.772+0000 7fca2a260000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-01-11T14:00:00.843 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:00.845+0000 7fca2a260000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-01-11T14:00:00.982 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:00.984+0000 7fca2a260000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-01-11T14:00:01.374 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:01.376+0000 7fca2a260000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-01-11T14:00:01.480 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:01.482+0000 7fca2a260000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-01-11T14:00:02.245 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:02.247+0000 7fca2a260000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-01-11T14:00:02.318 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:02.320+0000 7fca2a260000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-01-11T14:00:02.390 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:02.391+0000 7fca2a260000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-01-11T14:00:02.514 INFO:tasks.ceph.c1.osd.2.smithi161.stderr:2022-01-11T14:00:02.516+0000 7f03885b1240 -1 osd.2 0 log_to_monitors true 2022-01-11T14:00:02.532 INFO:tasks.ceph.c1.osd.0.smithi161.stderr:2022-01-11T14:00:02.534+0000 7f8c5a09c240 -1 osd.0 0 log_to_monitors true 2022-01-11T14:00:02.533 INFO:tasks.ceph.c1.osd.1.smithi161.stderr:2022-01-11T14:00:02.534+0000 7f719e189240 -1 osd.1 0 log_to_monitors true 2022-01-11T14:00:03.047 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:03.048+0000 7fca2a260000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-01-11T14:00:03.120 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:03.121+0000 7fca2a260000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-01-11T14:00:03.304 INFO:tasks.ceph.c1.mgr.x.smithi161.stderr:2022-01-11T14:00:03.306+0000 7fca2a260000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-01-11T14:00:03.653 INFO:tasks.ceph.c1.osd.1.smithi161.stderr:2022-01-11T14:00:03.655+0000 7f7194dab700 -1 osd.1 0 waiting for initial osdmap 2022-01-11T14:00:03.654 INFO:tasks.ceph.c1.osd.2.smithi161.stderr:2022-01-11T14:00:03.656+0000 7f037f1d3700 -1 osd.2 0 waiting for initial osdmap 2022-01-11T14:00:03.656 INFO:tasks.ceph.c1.osd.1.smithi161.stderr:2022-01-11T14:00:03.658+0000 7f718d73d700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-01-11T14:00:03.656 INFO:tasks.ceph.c1.osd.0.smithi161.stderr:2022-01-11T14:00:03.658+0000 7f8c50cbe700 -1 osd.0 0 waiting for initial osdmap 2022-01-11T14:00:03.658 INFO:tasks.ceph.c1.osd.2.smithi161.stderr:2022-01-11T14:00:03.659+0000 7f0377b65700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-01-11T14:00:03.663 INFO:tasks.ceph.c1.osd.0.smithi161.stderr:2022-01-11T14:00:03.665+0000 7f8c49650700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-01-11T14:00:06.378 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2022-01-11T14:00:06.673 INFO:teuthology.misc.health.smithi161.stdout: 2022-01-11T14:00:06.674 INFO:teuthology.misc.health.smithi161.stdout:{"epoch":9,"fsid":"fd8a9b68-e58b-4f3b-8917-54d7aadd438b","created":"2022-01-11T13:59:58.214991+0000","modified":"2022-01-11T14:00:06.330345+0000","last_up_change":"2022-01-11T14:00:04.657869+0000","last_in_change":"2022-01-11T13:59:59.145902+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-01-11T14:00:05.367242+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"f27f63ff-7822-47ef-9fc6-46ca314c3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":32000},{"type":"v1","addr":"172.21.15.161:6810","nonce":32000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":32000},{"type":"v1","addr":"172.21.15.161:6814","nonce":32000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6821","nonce":32000},{"type":"v1","addr":"172.21.15.161:6823","nonce":32000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":32000},{"type":"v1","addr":"172.21.15.161:6818","nonce":32000}]},"public_addr":"172.21.15.161:6810/32000","cluster_addr":"172.21.15.161:6814/32000","heartbeat_back_addr":"172.21.15.161:6823/32000","heartbeat_front_addr":"172.21.15.161:6818/32000","state":["exists","up"]},{"osd":1,"uuid":"17448e5a-bf40-441c-b631-c071795fe273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":31999},{"type":"v1","addr":"172.21.15.161:6801","nonce":31999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":31999},{"type":"v1","addr":"172.21.15.161:6803","nonce":31999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":31999},{"type":"v1","addr":"172.21.15.161:6807","nonce":31999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":31999},{"type":"v1","addr":"172.21.15.161:6805","nonce":31999}]},"public_addr":"172.21.15.161:6801/31999","cluster_addr":"172.21.15.161:6803/31999","heartbeat_back_addr":"172.21.15.161:6807/31999","heartbeat_front_addr":"172.21.15.161:6805/31999","state":["exists","up"]},{"osd":2,"uuid":"2b6eb01f-1796-4534-935e-0f194d39b3cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":32001},{"type":"v1","addr":"172.21.15.161:6811","nonce":32001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6813","nonce":32001},{"type":"v1","addr":"172.21.15.161:6815","nonce":32001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":32001},{"type":"v1","addr":"172.21.15.161:6822","nonce":32001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6817","nonce":32001},{"type":"v1","addr":"172.21.15.161:6819","nonce":32001}]},"public_addr":"172.21.15.161:6811/32001","cluster_addr":"172.21.15.161:6815/32001","heartbeat_back_addr":"172.21.15.161:6822/32001","heartbeat_front_addr":"172.21.15.161:6819/32001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:03.538597+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:03.486811+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:03.542943+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-11T14:00:06.686 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-01-11T14:00:06.687 INFO:tasks.ceph:Creating RBD pool 2022-01-11T14:00:06.688 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster c1 osd pool create rbd 8 2022-01-11T14:00:07.342 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'rbd' created 2022-01-11T14:00:07.351 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2022-01-11T14:00:08.342 INFO:teuthology.orchestra.run.smithi161.stderr:enabled application 'rbd' on pool 'rbd' 2022-01-11T14:00:08.354 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2022-01-11T14:00:08.354 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2022-01-11T14:00:08.355 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2022-01-11T14:00:08.356 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2022-01-11T14:00:08.357 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-01-11T14:00:08.702 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:08.714 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":31662},{"type":"v1","addr":"172.21.15.161:6825","nonce":31662}]},"active_addr":"172.21.15.161:6825/31662","active_change":"2022-01-11T14:00:03.320068+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":1255827085}]},{"addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":2240899193}]},{"addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":3612676079}]}]}} 2022-01-11T14:00:08.717 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2022-01-11T14:00:08.717 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2022-01-11T14:00:08.718 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-01-11T14:00:09.011 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:09.012 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":11,"fsid":"fd8a9b68-e58b-4f3b-8917-54d7aadd438b","created":"2022-01-11T13:59:58.214991+0000","modified":"2022-01-11T14:00:08.336632+0000","last_up_change":"2022-01-11T14:00:04.657869+0000","last_in_change":"2022-01-11T13:59:59.145902+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-01-11T14:00:05.367242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-01-11T14:00:07.002484+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f27f63ff-7822-47ef-9fc6-46ca314c3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":32000},{"type":"v1","addr":"172.21.15.161:6810","nonce":32000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":32000},{"type":"v1","addr":"172.21.15.161:6814","nonce":32000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6821","nonce":32000},{"type":"v1","addr":"172.21.15.161:6823","nonce":32000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":32000},{"type":"v1","addr":"172.21.15.161:6818","nonce":32000}]},"public_addr":"172.21.15.161:6810/32000","cluster_addr":"172.21.15.161:6814/32000","heartbeat_back_addr":"172.21.15.161:6823/32000","heartbeat_front_addr":"172.21.15.161:6818/32000","state":["exists","up"]},{"osd":1,"uuid":"17448e5a-bf40-441c-b631-c071795fe273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":31999},{"type":"v1","addr":"172.21.15.161:6801","nonce":31999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":31999},{"type":"v1","addr":"172.21.15.161:6803","nonce":31999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":31999},{"type":"v1","addr":"172.21.15.161:6807","nonce":31999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":31999},{"type":"v1","addr":"172.21.15.161:6805","nonce":31999}]},"public_addr":"172.21.15.161:6801/31999","cluster_addr":"172.21.15.161:6803/31999","heartbeat_back_addr":"172.21.15.161:6807/31999","heartbeat_front_addr":"172.21.15.161:6805/31999","state":["exists","up"]},{"osd":2,"uuid":"2b6eb01f-1796-4534-935e-0f194d39b3cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":32001},{"type":"v1","addr":"172.21.15.161:6811","nonce":32001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6813","nonce":32001},{"type":"v1","addr":"172.21.15.161:6815","nonce":32001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":32001},{"type":"v1","addr":"172.21.15.161:6822","nonce":32001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6817","nonce":32001},{"type":"v1","addr":"172.21.15.161:6819","nonce":32001}]},"public_addr":"172.21.15.161:6811/32001","cluster_addr":"172.21.15.161:6815/32001","heartbeat_back_addr":"172.21.15.161:6822/32001","heartbeat_front_addr":"172.21.15.161:6819/32001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:03.538597+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:03.486811+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:03.542943+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-11T14:00:09.024 INFO:tasks.ceph.ceph_manager.c1:all up! 2022-01-11T14:00:09.025 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2022-01-11T14:00:09.316 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:09.317 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":11,"fsid":"fd8a9b68-e58b-4f3b-8917-54d7aadd438b","created":"2022-01-11T13:59:58.214991+0000","modified":"2022-01-11T14:00:08.336632+0000","last_up_change":"2022-01-11T14:00:04.657869+0000","last_in_change":"2022-01-11T13:59:59.145902+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-01-11T14:00:05.367242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-01-11T14:00:07.002484+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f27f63ff-7822-47ef-9fc6-46ca314c3052","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":32000},{"type":"v1","addr":"172.21.15.161:6810","nonce":32000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":32000},{"type":"v1","addr":"172.21.15.161:6814","nonce":32000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6821","nonce":32000},{"type":"v1","addr":"172.21.15.161:6823","nonce":32000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":32000},{"type":"v1","addr":"172.21.15.161:6818","nonce":32000}]},"public_addr":"172.21.15.161:6810/32000","cluster_addr":"172.21.15.161:6814/32000","heartbeat_back_addr":"172.21.15.161:6823/32000","heartbeat_front_addr":"172.21.15.161:6818/32000","state":["exists","up"]},{"osd":1,"uuid":"17448e5a-bf40-441c-b631-c071795fe273","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":31999},{"type":"v1","addr":"172.21.15.161:6801","nonce":31999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":31999},{"type":"v1","addr":"172.21.15.161:6803","nonce":31999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":31999},{"type":"v1","addr":"172.21.15.161:6807","nonce":31999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":31999},{"type":"v1","addr":"172.21.15.161:6805","nonce":31999}]},"public_addr":"172.21.15.161:6801/31999","cluster_addr":"172.21.15.161:6803/31999","heartbeat_back_addr":"172.21.15.161:6807/31999","heartbeat_front_addr":"172.21.15.161:6805/31999","state":["exists","up"]},{"osd":2,"uuid":"2b6eb01f-1796-4534-935e-0f194d39b3cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":32001},{"type":"v1","addr":"172.21.15.161:6811","nonce":32001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6813","nonce":32001},{"type":"v1","addr":"172.21.15.161:6815","nonce":32001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":32001},{"type":"v1","addr":"172.21.15.161:6822","nonce":32001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6817","nonce":32001},{"type":"v1","addr":"172.21.15.161:6819","nonce":32001}]},"public_addr":"172.21.15.161:6811/32001","cluster_addr":"172.21.15.161:6815/32001","heartbeat_back_addr":"172.21.15.161:6822/32001","heartbeat_front_addr":"172.21.15.161:6819/32001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:03.538597+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:03.486811+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:03.542943+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-11T14:00:09.330 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2022-01-11T14:00:09.331 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2022-01-11T14:00:09.331 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2022-01-11T14:00:09.494 INFO:teuthology.orchestra.run.smithi161.stdout:34359738370 2022-01-11T14:00:09.494 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-01-11T14:00:09.500 INFO:teuthology.orchestra.run.smithi161.stdout:34359738370 2022-01-11T14:00:09.500 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2022-01-11T14:00:09.508 INFO:teuthology.orchestra.run.smithi161.stdout:34359738371 2022-01-11T14:00:09.508 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-01-11T14:00:09.805 INFO:teuthology.orchestra.run.smithi161.stdout:34359738369 2022-01-11T14:00:09.818 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.0 2022-01-11T14:00:09.857 INFO:teuthology.orchestra.run.smithi161.stdout:34359738370 2022-01-11T14:00:09.858 INFO:teuthology.orchestra.run.smithi161.stdout:34359738369 2022-01-11T14:00:09.871 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.2 2022-01-11T14:00:09.872 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.1 2022-01-11T14:00:10.820 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-01-11T14:00:10.873 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-01-11T14:00:10.874 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2022-01-11T14:00:11.150 INFO:teuthology.orchestra.run.smithi161.stdout:34359738369 2022-01-11T14:00:11.164 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.0 2022-01-11T14:00:11.190 INFO:teuthology.orchestra.run.smithi161.stdout:34359738369 2022-01-11T14:00:11.198 INFO:teuthology.orchestra.run.smithi161.stdout:34359738370 2022-01-11T14:00:11.204 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.1 2022-01-11T14:00:11.213 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.2 2022-01-11T14:00:12.166 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2022-01-11T14:00:12.205 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2022-01-11T14:00:12.214 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2022-01-11T14:00:12.496 INFO:teuthology.orchestra.run.smithi161.stdout:34359738371 2022-01-11T14:00:12.510 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738371 for osd.0 2022-01-11T14:00:12.511 DEBUG:teuthology.parallel:result is None 2022-01-11T14:00:12.558 INFO:teuthology.orchestra.run.smithi161.stdout:34359738371 2022-01-11T14:00:12.571 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738371 for osd.1 2022-01-11T14:00:12.572 DEBUG:teuthology.parallel:result is None 2022-01-11T14:00:12.592 INFO:teuthology.orchestra.run.smithi161.stdout:34359738372 2022-01-11T14:00:12.604 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738372 for osd.2 2022-01-11T14:00:12.605 DEBUG:teuthology.parallel:result is None 2022-01-11T14:00:12.605 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2022-01-11T14:00:12.606 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2022-01-11T14:00:12.932 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:12.934 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2022-01-11T14:00:12.945 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2022-01-11T14:00:11.327107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":17812,"kb_used_data":1212,"kb_used_omap":0,"kb_used_meta":16320,"kb_avail":314554988,"statfs":{"total":322122547200,"available":322104307712,"internally_reserved":0,"allocated":1241088,"data_stored":1013432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16711680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.345203+0000","last_change":"2022-01-11T14:00:08.343593+0000","last_active":"2022-01-11T14:00:08.345203+0000","last_peered":"2022-01-11T14:00:08.345203+0000","last_clean":"2022-01-11T14:00:08.345203+0000","last_became_active":"2022-01-11T14:00:08.343330+0000","last_became_peered":"2022-01-11T14:00:08.343330+0000","last_unstale":"2022-01-11T14:00:08.345203+0000","last_undegraded":"2022-01-11T14:00:08.345203+0000","last_fullsized":"2022-01-11T14:00:08.345203+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T22:58:46.492145+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.344409+0000","last_change":"2022-01-11T14:00:08.342869+0000","last_active":"2022-01-11T14:00:08.344409+0000","last_peered":"2022-01-11T14:00:08.344409+0000","last_clean":"2022-01-11T14:00:08.344409+0000","last_became_active":"2022-01-11T14:00:08.342607+0000","last_became_peered":"2022-01-11T14:00:08.342607+0000","last_unstale":"2022-01-11T14:00:08.344409+0000","last_undegraded":"2022-01-11T14:00:08.344409+0000","last_fullsized":"2022-01-11T14:00:08.344409+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T23:13:20.218849+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.344406+0000","last_change":"2022-01-11T14:00:08.342596+0000","last_active":"2022-01-11T14:00:08.344406+0000","last_peered":"2022-01-11T14:00:08.344406+0000","last_clean":"2022-01-11T14:00:08.344406+0000","last_became_active":"2022-01-11T14:00:08.342050+0000","last_became_peered":"2022-01-11T14:00:08.342050+0000","last_unstale":"2022-01-11T14:00:08.344406+0000","last_undegraded":"2022-01-11T14:00:08.344406+0000","last_fullsized":"2022-01-11T14:00:08.344406+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T23:53:13.380220+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.345232+0000","last_change":"2022-01-11T14:00:08.344257+0000","last_active":"2022-01-11T14:00:08.345232+0000","last_peered":"2022-01-11T14:00:08.345232+0000","last_clean":"2022-01-11T14:00:08.345232+0000","last_became_active":"2022-01-11T14:00:08.343991+0000","last_became_peered":"2022-01-11T14:00:08.343991+0000","last_unstale":"2022-01-11T14:00:08.345232+0000","last_undegraded":"2022-01-11T14:00:08.345232+0000","last_fullsized":"2022-01-11T14:00:08.345232+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T22:27:44.876191+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.338550+0000","last_change":"2022-01-11T14:00:07.342112+0000","last_active":"2022-01-11T14:00:08.338550+0000","last_peered":"2022-01-11T14:00:08.338550+0000","last_clean":"2022-01-11T14:00:08.338550+0000","last_became_active":"2022-01-11T14:00:07.341838+0000","last_became_peered":"2022-01-11T14:00:07.341838+0000","last_unstale":"2022-01-11T14:00:08.338550+0000","last_undegraded":"2022-01-11T14:00:08.338550+0000","last_fullsized":"2022-01-11T14:00:08.338550+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:06.330345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:06.330345+0000","last_clean_scrub_stamp":"2022-01-11T14:00:06.330345+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T23:57:44.410337+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.343756+0000","last_change":"2022-01-11T14:00:08.342109+0000","last_active":"2022-01-11T14:00:08.343756+0000","last_peered":"2022-01-11T14:00:08.343756+0000","last_clean":"2022-01-11T14:00:08.343756+0000","last_became_active":"2022-01-11T14:00:08.341855+0000","last_became_peered":"2022-01-11T14:00:08.341855+0000","last_unstale":"2022-01-11T14:00:08.343756+0000","last_undegraded":"2022-01-11T14:00:08.343756+0000","last_fullsized":"2022-01-11T14:00:08.343756+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T23:45:08.718818+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.346013+0000","last_change":"2022-01-11T14:00:08.344376+0000","last_active":"2022-01-11T14:00:08.346013+0000","last_peered":"2022-01-11T14:00:08.346013+0000","last_clean":"2022-01-11T14:00:08.346013+0000","last_became_active":"2022-01-11T14:00:08.344104+0000","last_became_peered":"2022-01-11T14:00:08.344104+0000","last_unstale":"2022-01-11T14:00:08.346013+0000","last_undegraded":"2022-01-11T14:00:08.346013+0000","last_fullsized":"2022-01-11T14:00:08.346013+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T19:41:17.610301+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.345704+0000","last_change":"2022-01-11T14:00:08.344926+0000","last_active":"2022-01-11T14:00:08.345704+0000","last_peered":"2022-01-11T14:00:08.345704+0000","last_clean":"2022-01-11T14:00:08.345704+0000","last_became_active":"2022-01-11T14:00:08.344645+0000","last_became_peered":"2022-01-11T14:00:08.344645+0000","last_unstale":"2022-01-11T14:00:08.345704+0000","last_undegraded":"2022-01-11T14:00:08.345704+0000","last_fullsized":"2022-01-11T14:00:08.345704+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T17:06:49.757911+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.344398+0000","last_change":"2022-01-11T14:00:08.342850+0000","last_active":"2022-01-11T14:00:08.344398+0000","last_peered":"2022-01-11T14:00:08.344398+0000","last_clean":"2022-01-11T14:00:08.344398+0000","last_became_active":"2022-01-11T14:00:08.342669+0000","last_became_peered":"2022-01-11T14:00:08.342669+0000","last_unstale":"2022-01-11T14:00:08.344398+0000","last_undegraded":"2022-01-11T14:00:08.344398+0000","last_fullsized":"2022-01-11T14:00:08.344398+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T17:18:49.024961+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5296,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":104852304,"statfs":{"total":107374182400,"available":107368759296,"internally_reserved":0,"allocated":106496,"data_stored":29071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":6256,"kb_used_data":552,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":104851344,"statfs":{"total":107374182400,"available":107367776256,"internally_reserved":0,"allocated":565248,"data_stored":491800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":6260,"kb_used_data":556,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":104851340,"statfs":{"total":107374182400,"available":107367772160,"internally_reserved":0,"allocated":569344,"data_stored":492561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-01-11T14:00:12.946 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2022-01-11T14:00:13.235 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:13.237 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2022-01-11T14:00:13.248 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2022-01-11T14:00:11.327107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":17812,"kb_used_data":1212,"kb_used_omap":0,"kb_used_meta":16320,"kb_avail":314554988,"statfs":{"total":322122547200,"available":322104307712,"internally_reserved":0,"allocated":1241088,"data_stored":1013432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16711680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.345203+0000","last_change":"2022-01-11T14:00:08.343593+0000","last_active":"2022-01-11T14:00:08.345203+0000","last_peered":"2022-01-11T14:00:08.345203+0000","last_clean":"2022-01-11T14:00:08.345203+0000","last_became_active":"2022-01-11T14:00:08.343330+0000","last_became_peered":"2022-01-11T14:00:08.343330+0000","last_unstale":"2022-01-11T14:00:08.345203+0000","last_undegraded":"2022-01-11T14:00:08.345203+0000","last_fullsized":"2022-01-11T14:00:08.345203+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T22:58:46.492145+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.344409+0000","last_change":"2022-01-11T14:00:08.342869+0000","last_active":"2022-01-11T14:00:08.344409+0000","last_peered":"2022-01-11T14:00:08.344409+0000","last_clean":"2022-01-11T14:00:08.344409+0000","last_became_active":"2022-01-11T14:00:08.342607+0000","last_became_peered":"2022-01-11T14:00:08.342607+0000","last_unstale":"2022-01-11T14:00:08.344409+0000","last_undegraded":"2022-01-11T14:00:08.344409+0000","last_fullsized":"2022-01-11T14:00:08.344409+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T23:13:20.218849+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.344406+0000","last_change":"2022-01-11T14:00:08.342596+0000","last_active":"2022-01-11T14:00:08.344406+0000","last_peered":"2022-01-11T14:00:08.344406+0000","last_clean":"2022-01-11T14:00:08.344406+0000","last_became_active":"2022-01-11T14:00:08.342050+0000","last_became_peered":"2022-01-11T14:00:08.342050+0000","last_unstale":"2022-01-11T14:00:08.344406+0000","last_undegraded":"2022-01-11T14:00:08.344406+0000","last_fullsized":"2022-01-11T14:00:08.344406+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T23:53:13.380220+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.345232+0000","last_change":"2022-01-11T14:00:08.344257+0000","last_active":"2022-01-11T14:00:08.345232+0000","last_peered":"2022-01-11T14:00:08.345232+0000","last_clean":"2022-01-11T14:00:08.345232+0000","last_became_active":"2022-01-11T14:00:08.343991+0000","last_became_peered":"2022-01-11T14:00:08.343991+0000","last_unstale":"2022-01-11T14:00:08.345232+0000","last_undegraded":"2022-01-11T14:00:08.345232+0000","last_fullsized":"2022-01-11T14:00:08.345232+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T22:27:44.876191+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.338550+0000","last_change":"2022-01-11T14:00:07.342112+0000","last_active":"2022-01-11T14:00:08.338550+0000","last_peered":"2022-01-11T14:00:08.338550+0000","last_clean":"2022-01-11T14:00:08.338550+0000","last_became_active":"2022-01-11T14:00:07.341838+0000","last_became_peered":"2022-01-11T14:00:07.341838+0000","last_unstale":"2022-01-11T14:00:08.338550+0000","last_undegraded":"2022-01-11T14:00:08.338550+0000","last_fullsized":"2022-01-11T14:00:08.338550+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:06.330345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:06.330345+0000","last_clean_scrub_stamp":"2022-01-11T14:00:06.330345+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T23:57:44.410337+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.343756+0000","last_change":"2022-01-11T14:00:08.342109+0000","last_active":"2022-01-11T14:00:08.343756+0000","last_peered":"2022-01-11T14:00:08.343756+0000","last_clean":"2022-01-11T14:00:08.343756+0000","last_became_active":"2022-01-11T14:00:08.341855+0000","last_became_peered":"2022-01-11T14:00:08.341855+0000","last_unstale":"2022-01-11T14:00:08.343756+0000","last_undegraded":"2022-01-11T14:00:08.343756+0000","last_fullsized":"2022-01-11T14:00:08.343756+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T23:45:08.718818+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.346013+0000","last_change":"2022-01-11T14:00:08.344376+0000","last_active":"2022-01-11T14:00:08.346013+0000","last_peered":"2022-01-11T14:00:08.346013+0000","last_clean":"2022-01-11T14:00:08.346013+0000","last_became_active":"2022-01-11T14:00:08.344104+0000","last_became_peered":"2022-01-11T14:00:08.344104+0000","last_unstale":"2022-01-11T14:00:08.346013+0000","last_undegraded":"2022-01-11T14:00:08.346013+0000","last_fullsized":"2022-01-11T14:00:08.346013+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T19:41:17.610301+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.345704+0000","last_change":"2022-01-11T14:00:08.344926+0000","last_active":"2022-01-11T14:00:08.345704+0000","last_peered":"2022-01-11T14:00:08.345704+0000","last_clean":"2022-01-11T14:00:08.345704+0000","last_became_active":"2022-01-11T14:00:08.344645+0000","last_became_peered":"2022-01-11T14:00:08.344645+0000","last_unstale":"2022-01-11T14:00:08.345704+0000","last_undegraded":"2022-01-11T14:00:08.345704+0000","last_fullsized":"2022-01-11T14:00:08.345704+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T17:06:49.757911+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:08.344398+0000","last_change":"2022-01-11T14:00:08.342850+0000","last_active":"2022-01-11T14:00:08.344398+0000","last_peered":"2022-01-11T14:00:08.344398+0000","last_clean":"2022-01-11T14:00:08.344398+0000","last_became_active":"2022-01-11T14:00:08.342669+0000","last_became_peered":"2022-01-11T14:00:08.342669+0000","last_unstale":"2022-01-11T14:00:08.344398+0000","last_undegraded":"2022-01-11T14:00:08.344398+0000","last_fullsized":"2022-01-11T14:00:08.344398+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:07.334540+0000","last_clean_scrub_stamp":"2022-01-11T14:00:07.334540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T17:18:49.024961+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5296,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":104852304,"statfs":{"total":107374182400,"available":107368759296,"internally_reserved":0,"allocated":106496,"data_stored":29071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":6256,"kb_used_data":552,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":104851344,"statfs":{"total":107374182400,"available":107367776256,"internally_reserved":0,"allocated":565248,"data_stored":491800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":6260,"kb_used_data":556,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":104851340,"statfs":{"total":107374182400,"available":107367772160,"internally_reserved":0,"allocated":569344,"data_stored":492561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-01-11T14:00:13.249 INFO:tasks.ceph.ceph_manager.c1:clean! 2022-01-11T14:00:13.250 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2022-01-11T14:00:13.250 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2022-01-11T14:00:13.250 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2022-01-11T14:00:13.581 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:13.582 INFO:teuthology.orchestra.run.smithi161.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-01-11T14:00:13.595 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2022-01-11T14:00:13.595 INFO:teuthology.run_tasks:Running task ceph... 2022-01-11T14:00:13.605 INFO:tasks.ceph:Creating ceph cluster c2... 2022-01-11T14:00:13.606 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-01-11T14:00:13.606 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dang-2022-01-11_08:37:46-rgw-wip-dang-testing-distro-basic-smithi/6608060', 'branch': 'wip-dang-testing', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '6608060', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'dang-2022-01-11_08:37:46-rgw-wip-dang-testing-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.3', 'overrides': {'admin_socket': {'branch': 'wip-dang-testing'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '17f142526478a11e8f62d3933be552d546c4fd19', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '17f142526478a11e8f62d3933be552d546c4fd19'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-dang-testing', 'sha1': '17f142526478a11e8f62d3933be552d546c4fd19'}}, 'owner': 'scheduled_dang@sidious.arb.redhat.com', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '17f142526478a11e8f62d3933be552d546c4fd19', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-dang-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_17f142526478a11e8f62d3933be552d546c4fd19/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '17f142526478a11e8f62d3933be552d546c4fd19', 'targets': {'smithi161.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+BHf5FlXf0gP7b+a0ejm3R2ApbRE9mcCiTGRp/GWnfGX07cnqSVzw94iCpFRzX6J3cMV8d+NLP9mYK1TQLZTixMkl+OGh+1b6hHxy/MgCPBVK3z8hNjD3cVknHoa3bIs3sYw54hPJV/toQO705SfrzsHeMi4x5339qeAxOR2bnU3KjUI4UJ046tIg5q+DfWZNm7YTOna3Ho0amBKCk5qydCf6LG1ZJP5vYjhGadjW1X9h0r+OhsMaVYf4p6CI7SEDEAy+ZDStL8pwDEes8SAXagWUe39jnXq7hLdg1R9O9Z7Vqf4v2NpTv1sqP6ViqY3fq7rGyYxdZEfw0K/sPufQLlGoW5Vt/Uevi+iu1z5p0pObvUsJwDvQ3QiEzh9bmTQ3BY3nxssCWYRzV4G5iYoUGfUQ3df0bI2tLxQ6Ox38sZUHtFCe5wWkFPh8woIlGhrp9ujJkciUSt+9QtHcfq9O+fw4v0LMBftbSlJK+nYwxGpx8ofcfKk65QS/sFiGDo0=', 'smithi172.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJy9sSSPAjBB6dmli0gWyyA/2N0akOcMO4MO8j5C90Y0c+2I7/6VsrqpueZds/gd9DvL83u49CBfO8h3Zjl5a9HRntGaDGZAqj8DM+3y/1/G2LzgJ2j5WUxIHcdJlRfNWxmGp3iQO0+Dnx0NtQY/JE94GEoLgI+3C8CGaxPnLDQWuNHK+1OMEHUGOppA41xNRzPm7NwEWKEtAh8sa9Fx3VaCXHSqvdxjY1KFg5xSvKkpCF5Wkt60pb8w0GHwM4HqtbbBOwuDIbwQa3cH4ZnriHqypTyVg48xmBOjXjLeClyxete2ExIxSgjlKigQYdU9PKXGO7D+wFunduTokQSA5X2IRH30cy3UvFZyYZpoPHymZCPg+ZJMHic12xNdA36MiSxFOdE/y3ZkQ7Apd+Sz4D6W7+RdBXBPHu67bWf6GCh4WJdCXs1WKAaIimt0SHeNCmYl+jgWy04IgGkm6fd4lGD0VMs+Ub/qj/P8pmyGEzjmizhoBpXpl//Od+07hBfh8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '17f142526478a11e8f62d3933be552d546c4fd19', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '17f142526478a11e8f62d3933be552d546c4fd19', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '00e09f05e9fb2ab7c7f497f5e446613531743ae2', 'timestamp': '2022-01-11_08:37:46', 'tube': 'smithi', 'user': 'dang', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.20261'} 2022-01-11T14:00:13.607 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2022-01-11T14:00:13.610 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2022-01-11T14:00:13.628 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/run/ceph 2022-01-11T14:00:13.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2022-01-11T14:00:13.699 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:13.699 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2022-01-11T14:00:13.755 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-01-11T14:00:13.756 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-01-11T14:00:13.812 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-01-11T14:00:13.812 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-11T14:00:13.812 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 108286 Links: 1 2022-01-11T14:00:13.813 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-11T14:00:13.813 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2022-01-11T14:00:13.813 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2022-01-11 13:59:42.268629044 +0000 2022-01-11T14:00:13.813 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2022-01-11 13:56:23.557998714 +0000 2022-01-11T14:00:13.814 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2022-01-11 13:56:23.557998714 +0000 2022-01-11T14:00:13.814 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2022-01-11T14:00:13.823 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2022-01-11T14:00:13.824 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2022-01-11T14:00:13.824 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000251844 s, 2.0 MB/s 2022-01-11T14:00:13.827 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-01-11T14:00:13.884 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-01-11T14:00:13.884 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-11T14:00:13.885 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 112153 Links: 1 2022-01-11T14:00:13.885 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-11T14:00:13.885 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2022-01-11T14:00:13.885 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2022-01-11 13:59:42.268629044 +0000 2022-01-11T14:00:13.886 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2022-01-11 13:56:23.906990323 +0000 2022-01-11T14:00:13.886 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2022-01-11 13:56:23.906990323 +0000 2022-01-11T14:00:13.886 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2022-01-11T14:00:13.894 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2022-01-11T14:00:13.895 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2022-01-11T14:00:13.895 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000225343 s, 2.3 MB/s 2022-01-11T14:00:13.897 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-01-11T14:00:13.953 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-01-11T14:00:13.953 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-11T14:00:13.954 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 113256 Links: 1 2022-01-11T14:00:13.954 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-11T14:00:13.954 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2022-01-11T14:00:13.955 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2022-01-11 13:59:42.268629044 +0000 2022-01-11T14:00:13.955 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2022-01-11 13:56:24.224982679 +0000 2022-01-11T14:00:13.955 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2022-01-11 13:56:24.224982679 +0000 2022-01-11T14:00:13.955 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2022-01-11T14:00:13.964 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2022-01-11T14:00:13.964 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2022-01-11T14:00:13.964 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000215964 s, 2.4 MB/s 2022-01-11T14:00:13.967 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-01-11T14:00:14.023 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-01-11T14:00:14.023 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-01-11T14:00:14.023 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 110222 Links: 1 2022-01-11T14:00:14.024 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-01-11T14:00:14.024 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2022-01-11T14:00:14.024 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2022-01-11 13:59:42.268629044 +0000 2022-01-11T14:00:14.024 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2022-01-11 13:56:24.564974504 +0000 2022-01-11T14:00:14.025 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2022-01-11 13:56:24.564974504 +0000 2022-01-11T14:00:14.025 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2022-01-11T14:00:14.033 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2022-01-11T14:00:14.034 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2022-01-11T14:00:14.034 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000201721 s, 2.5 MB/s 2022-01-11T14:00:14.036 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-01-11T14:00:14.036 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2022-01-11T14:00:14.037 INFO:tasks.ceph:Generating config... 2022-01-11T14:00:14.038 INFO:tasks.ceph:[client] debug rgw = 20 2022-01-11T14:00:14.039 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2022-01-11T14:00:14.039 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2022-01-11T14:00:14.039 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-01-11T14:00:14.039 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2022-01-11T14:00:14.040 INFO:tasks.ceph:[client] rgw data log num shards = 4 2022-01-11T14:00:14.040 INFO:tasks.ceph:[client] rgw md log max shards = 4 2022-01-11T14:00:14.040 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2022-01-11T14:00:14.041 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2022-01-11T14:00:14.041 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2022-01-11T14:00:14.041 INFO:tasks.ceph:[client] setgroup = ceph 2022-01-11T14:00:14.042 INFO:tasks.ceph:[client] setuser = ceph 2022-01-11T14:00:14.042 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-01-11T14:00:14.042 INFO:tasks.ceph:[mgr] debug ms = 1 2022-01-11T14:00:14.042 INFO:tasks.ceph:[mon] debug mon = 20 2022-01-11T14:00:14.043 INFO:tasks.ceph:[mon] debug ms = 1 2022-01-11T14:00:14.043 INFO:tasks.ceph:[mon] debug paxos = 20 2022-01-11T14:00:14.043 INFO:tasks.ceph:[osd] debug ms = 1 2022-01-11T14:00:14.044 INFO:tasks.ceph:[osd] debug osd = 20 2022-01-11T14:00:14.044 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2022-01-11T14:00:14.044 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2022-01-11T14:00:14.044 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2022-01-11T14:00:14.045 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2022-01-11T14:00:14.045 INFO:tasks.ceph:Setting up c2.mon.a... 2022-01-11T14:00:14.045 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2022-01-11T14:00:14.157 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/c2.keyring 2022-01-11T14:00:14.159 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2022-01-11T14:00:14.199 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-01-11T14:00:14.263 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.172')] 2022-01-11T14:00:14.264 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.172'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2022-01-11T14:00:14.265 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:14.265 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-01-11T14:00:14.318 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.172 --print /home/ubuntu/cephtest/c2.monmap 2022-01-11T14:00:14.393 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:00:14.394 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:00:14.395 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2022-01-11T14:00:14.396 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: generated fsid 65510500-97b1-4b61-b1ef-a0cc54b6a9d6 2022-01-11T14:00:14.396 INFO:teuthology.orchestra.run.smithi172.stdout:setting min_mon_release = octopus 2022-01-11T14:00:14.396 INFO:teuthology.orchestra.run.smithi172.stdout:epoch 0 2022-01-11T14:00:14.396 INFO:teuthology.orchestra.run.smithi172.stdout:fsid 65510500-97b1-4b61-b1ef-a0cc54b6a9d6 2022-01-11T14:00:14.396 INFO:teuthology.orchestra.run.smithi172.stdout:last_changed 2022-01-11T14:00:14.397332+0000 2022-01-11T14:00:14.397 INFO:teuthology.orchestra.run.smithi172.stdout:created 2022-01-11T14:00:14.397332+0000 2022-01-11T14:00:14.397 INFO:teuthology.orchestra.run.smithi172.stdout:min_mon_release 15 (octopus) 2022-01-11T14:00:14.397 INFO:teuthology.orchestra.run.smithi172.stdout:election_strategy: 1 2022-01-11T14:00:14.397 INFO:teuthology.orchestra.run.smithi172.stdout:0: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.a 2022-01-11T14:00:14.398 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2022-01-11T14:00:14.398 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-01-11T14:00:14.451 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 65510500-97b1-4b61-b1ef-a0cc54b6a9d6... 2022-01-11T14:00:14.452 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2022-01-11T14:00:14.455 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2022-01-11T14:00:14.499 INFO:teuthology.orchestra.run.smithi161.stdout:[global] 2022-01-11T14:00:14.499 INFO:teuthology.orchestra.run.smithi161.stdout: chdir = "" 2022-01-11T14:00:14.500 INFO:teuthology.orchestra.run.smithi161.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-01-11T14:00:14.500 INFO:teuthology.orchestra.run.smithi161.stdout: auth supported = cephx 2022-01-11T14:00:14.500 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.500 INFO:teuthology.orchestra.run.smithi161.stdout: filestore xattr use omap = true 2022-01-11T14:00:14.501 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.501 INFO:teuthology.orchestra.run.smithi161.stdout: mon clock drift allowed = 1.000 2022-01-11T14:00:14.501 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.501 INFO:teuthology.orchestra.run.smithi161.stdout: osd crush chooseleaf type = 0 2022-01-11T14:00:14.502 INFO:teuthology.orchestra.run.smithi161.stdout: auth debug = true 2022-01-11T14:00:14.502 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.502 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on old message = true 2022-01-11T14:00:14.502 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on bug = true 2022-01-11T14:00:14.502 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.503 INFO:teuthology.orchestra.run.smithi161.stdout: mon max pg per osd = 10000 # >= luminous 2022-01-11T14:00:14.503 INFO:teuthology.orchestra.run.smithi161.stdout: mon pg warn max object skew = 0 2022-01-11T14:00:14.503 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.503 INFO:teuthology.orchestra.run.smithi161.stdout: # disable pg_autoscaler by default for new pools 2022-01-11T14:00:14.504 INFO:teuthology.orchestra.run.smithi161.stdout: osd_pool_default_pg_autoscale_mode = off 2022-01-11T14:00:14.504 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.504 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default size = 2 2022-01-11T14:00:14.504 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.504 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow primary affinity = true 2022-01-11T14:00:14.505 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow pg remap = true 2022-01-11T14:00:14.505 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on legacy crush tunables = false 2022-01-11T14:00:14.505 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on crush straw calc version zero = false 2022-01-11T14:00:14.505 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on no sortbitwise = false 2022-01-11T14:00:14.506 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on osd down out interval zero = false 2022-01-11T14:00:14.506 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on too few osds = false 2022-01-11T14:00:14.506 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-01-11T14:00:14.506 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_no_redundancy = false 2022-01-11T14:00:14.507 INFO:teuthology.orchestra.run.smithi161.stdout: mon_allow_pool_size_one = true 2022-01-11T14:00:14.507 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.507 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-01-11T14:00:14.507 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.507 INFO:teuthology.orchestra.run.smithi161.stdout: osd default data pool replay window = 5 2022-01-11T14:00:14.508 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.508 INFO:teuthology.orchestra.run.smithi161.stdout: mon allow pool delete = true 2022-01-11T14:00:14.508 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.508 INFO:teuthology.orchestra.run.smithi161.stdout: mon cluster log file level = debug 2022-01-11T14:00:14.509 INFO:teuthology.orchestra.run.smithi161.stdout: debug asserts on shutdown = true 2022-01-11T14:00:14.509 INFO:teuthology.orchestra.run.smithi161.stdout: mon health detail to clog = false 2022-01-11T14:00:14.509 INFO:teuthology.orchestra.run.smithi161.stdout: mon host = 172.21.15.172 2022-01-11T14:00:14.509 INFO:teuthology.orchestra.run.smithi161.stdout: fsid = 65510500-97b1-4b61-b1ef-a0cc54b6a9d6 2022-01-11T14:00:14.509 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.510 INFO:teuthology.orchestra.run.smithi161.stdout:[osd] 2022-01-11T14:00:14.510 INFO:teuthology.orchestra.run.smithi161.stdout: osd journal size = 100 2022-01-11T14:00:14.510 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.511 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub load threshold = 5.0 2022-01-11T14:00:14.511 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub max interval = 600 2022-01-11T14:00:14.511 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.511 INFO:teuthology.orchestra.run.smithi161.stdout: osd recover clone overlap = true 2022-01-11T14:00:14.512 INFO:teuthology.orchestra.run.smithi161.stdout: osd recovery max chunk = 1048576 2022-01-11T14:00:14.512 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.512 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug shutdown = true 2022-01-11T14:00:14.512 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug op order = true 2022-01-11T14:00:14.513 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug verify stray on activate = true 2022-01-11T14:00:14.513 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.513 INFO:teuthology.orchestra.run.smithi161.stdout: osd open classes on start = true 2022-01-11T14:00:14.513 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug pg log writeout = true 2022-01-11T14:00:14.514 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.514 INFO:teuthology.orchestra.run.smithi161.stdout: osd deep scrub update digest min age = 30 2022-01-11T14:00:14.514 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.514 INFO:teuthology.orchestra.run.smithi161.stdout: osd map max advance = 10 2022-01-11T14:00:14.514 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.515 INFO:teuthology.orchestra.run.smithi161.stdout: journal zero on create = true 2022-01-11T14:00:14.515 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.515 INFO:teuthology.orchestra.run.smithi161.stdout: filestore ondisk finisher threads = 3 2022-01-11T14:00:14.515 INFO:teuthology.orchestra.run.smithi161.stdout: filestore apply finisher threads = 3 2022-01-11T14:00:14.516 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.516 INFO:teuthology.orchestra.run.smithi161.stdout: bdev debug aio = true 2022-01-11T14:00:14.516 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug misdirected ops = true 2022-01-11T14:00:14.516 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2022-01-11T14:00:14.517 INFO:teuthology.orchestra.run.smithi161.stdout: debug osd = 20 2022-01-11T14:00:14.517 INFO:teuthology.orchestra.run.smithi161.stdout: rocksdb delete range threshold = 0 2022-01-11T14:00:14.517 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.517 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr] 2022-01-11T14:00:14.517 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2022-01-11T14:00:14.518 INFO:teuthology.orchestra.run.smithi161.stdout: debug mgr = 20 2022-01-11T14:00:14.518 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2022-01-11T14:00:14.518 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2022-01-11T14:00:14.518 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2022-01-11T14:00:14.519 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2022-01-11T14:00:14.519 INFO:teuthology.orchestra.run.smithi161.stdout: mgr/telemetry/nag = false 2022-01-11T14:00:14.519 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.519 INFO:teuthology.orchestra.run.smithi161.stdout:[mon] 2022-01-11T14:00:14.521 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2022-01-11T14:00:14.521 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2022-01-11T14:00:14.521 INFO:teuthology.orchestra.run.smithi161.stdout: debug paxos = 20 2022-01-11T14:00:14.521 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2022-01-11T14:00:14.521 INFO:teuthology.orchestra.run.smithi161.stdout: mon data avail warn = 5 2022-01-11T14:00:14.522 INFO:teuthology.orchestra.run.smithi161.stdout: mon mgr mkfs grace = 240 2022-01-11T14:00:14.522 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2022-01-11T14:00:14.522 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd reporter subtree level = osd 2022-01-11T14:00:14.522 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd prime pg temp = true 2022-01-11T14:00:14.523 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2022-01-11T14:00:14.523 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.523 INFO:teuthology.orchestra.run.smithi161.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-01-11T14:00:14.523 INFO:teuthology.orchestra.run.smithi161.stdout: auth mon ticket ttl = 660 # 11m 2022-01-11T14:00:14.524 INFO:teuthology.orchestra.run.smithi161.stdout: auth service ticket ttl = 240 # 4m 2022-01-11T14:00:14.524 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.524 INFO:teuthology.orchestra.run.smithi161.stdout: # don't complain about insecure global_id in the test suite 2022-01-11T14:00:14.524 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-01-11T14:00:14.524 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-01-11T14:00:14.525 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.525 INFO:teuthology.orchestra.run.smithi161.stdout: # 1m isn't quite enough 2022-01-11T14:00:14.525 INFO:teuthology.orchestra.run.smithi161.stdout: mon_down_mkfs_grace = 2m 2022-01-11T14:00:14.525 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:00:14.526 INFO:teuthology.orchestra.run.smithi161.stdout:[client] 2022-01-11T14:00:14.526 INFO:teuthology.orchestra.run.smithi161.stdout: rgw cache enabled = true 2022-01-11T14:00:14.526 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable ops log = true 2022-01-11T14:00:14.526 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable usage log = true 2022-01-11T14:00:14.527 INFO:teuthology.orchestra.run.smithi161.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-01-11T14:00:14.527 INFO:teuthology.orchestra.run.smithi161.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-01-11T14:00:14.527 INFO:teuthology.orchestra.run.smithi161.stdout: debug rgw = 20 2022-01-11T14:00:14.527 INFO:teuthology.orchestra.run.smithi161.stdout: rgw crypt require ssl = False 2022-01-11T14:00:14.527 INFO:teuthology.orchestra.run.smithi161.stdout: rgw crypt s3 kms backend = testing 2022-01-11T14:00:14.528 INFO:teuthology.orchestra.run.smithi161.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-01-11T14:00:14.528 INFO:teuthology.orchestra.run.smithi161.stdout: rgw curl low speed time = 300 2022-01-11T14:00:14.528 INFO:teuthology.orchestra.run.smithi161.stdout: rgw data log num shards = 4 2022-01-11T14:00:14.528 INFO:teuthology.orchestra.run.smithi161.stdout: rgw md log max shards = 4 2022-01-11T14:00:14.529 INFO:teuthology.orchestra.run.smithi161.stdout: rgw sync log trim interval = 0 2022-01-11T14:00:14.529 INFO:teuthology.orchestra.run.smithi161.stdout: rgw sync meta inject err probability = 0.1 2022-01-11T14:00:14.529 INFO:teuthology.orchestra.run.smithi161.stdout: rgw sync obj etag verify = True 2022-01-11T14:00:14.529 INFO:teuthology.orchestra.run.smithi161.stdout: setgroup = ceph 2022-01-11T14:00:14.530 INFO:teuthology.orchestra.run.smithi161.stdout: setuser = ceph 2022-01-11T14:00:14.530 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.a] 2022-01-11T14:00:14.530 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.0] 2022-01-11T14:00:14.531 INFO:teuthology.orchestra.run.smithi161.stdout: osd_max_omap_entries_per_request = 10 2022-01-11T14:00:14.531 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.1] 2022-01-11T14:00:14.531 INFO:teuthology.orchestra.run.smithi161.stdout: osd_max_omap_entries_per_request = 1000 2022-01-11T14:00:14.531 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.2] 2022-01-11T14:00:14.531 INFO:teuthology.orchestra.run.smithi161.stdout: osd_max_omap_entries_per_request = 10000 2022-01-11T14:00:14.533 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2022-01-11T14:00:14.533 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2022-01-11T14:00:14.534 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-01-11T14:00:14.534 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2022-01-11T14:00:14.534 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.534 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2022-01-11T14:00:14.535 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.535 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2022-01-11T14:00:14.535 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.535 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2022-01-11T14:00:14.536 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2022-01-11T14:00:14.536 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.536 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2022-01-11T14:00:14.536 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2022-01-11T14:00:14.536 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.537 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2022-01-11T14:00:14.537 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2022-01-11T14:00:14.537 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.537 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2022-01-11T14:00:14.538 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2022-01-11T14:00:14.538 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.538 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2022-01-11T14:00:14.538 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.539 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2022-01-11T14:00:14.539 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2022-01-11T14:00:14.539 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2022-01-11T14:00:14.539 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2022-01-11T14:00:14.539 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2022-01-11T14:00:14.540 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2022-01-11T14:00:14.540 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2022-01-11T14:00:14.540 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-01-11T14:00:14.540 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2022-01-11T14:00:14.541 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2022-01-11T14:00:14.541 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.541 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-01-11T14:00:14.541 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.542 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2022-01-11T14:00:14.542 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.542 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2022-01-11T14:00:14.542 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.542 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2022-01-11T14:00:14.543 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2022-01-11T14:00:14.543 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2022-01-11T14:00:14.543 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = 172.21.15.172 2022-01-11T14:00:14.543 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 65510500-97b1-4b61-b1ef-a0cc54b6a9d6 2022-01-11T14:00:14.544 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.544 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2022-01-11T14:00:14.544 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2022-01-11T14:00:14.544 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.544 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2022-01-11T14:00:14.545 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2022-01-11T14:00:14.545 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.545 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2022-01-11T14:00:14.545 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2022-01-11T14:00:14.546 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.546 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2022-01-11T14:00:14.546 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2022-01-11T14:00:14.546 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2022-01-11T14:00:14.547 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.547 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2022-01-11T14:00:14.547 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2022-01-11T14:00:14.547 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.548 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2022-01-11T14:00:14.548 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.548 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2022-01-11T14:00:14.548 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.549 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2022-01-11T14:00:14.549 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.549 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2022-01-11T14:00:14.549 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2022-01-11T14:00:14.550 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.550 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2022-01-11T14:00:14.550 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2022-01-11T14:00:14.550 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2022-01-11T14:00:14.550 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2022-01-11T14:00:14.551 INFO:teuthology.orchestra.run.smithi172.stdout: rocksdb delete range threshold = 0 2022-01-11T14:00:14.551 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.551 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2022-01-11T14:00:14.551 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2022-01-11T14:00:14.552 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2022-01-11T14:00:14.552 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2022-01-11T14:00:14.552 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2022-01-11T14:00:14.552 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2022-01-11T14:00:14.553 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2022-01-11T14:00:14.553 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2022-01-11T14:00:14.553 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.553 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2022-01-11T14:00:14.553 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2022-01-11T14:00:14.554 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2022-01-11T14:00:14.554 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2022-01-11T14:00:14.554 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2022-01-11T14:00:14.554 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2022-01-11T14:00:14.555 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2022-01-11T14:00:14.555 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2022-01-11T14:00:14.555 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2022-01-11T14:00:14.555 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2022-01-11T14:00:14.556 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2022-01-11T14:00:14.556 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.556 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-01-11T14:00:14.556 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2022-01-11T14:00:14.557 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2022-01-11T14:00:14.557 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.557 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2022-01-11T14:00:14.557 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-01-11T14:00:14.557 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-01-11T14:00:14.558 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.558 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2022-01-11T14:00:14.558 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2022-01-11T14:00:14.558 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:14.559 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2022-01-11T14:00:14.559 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2022-01-11T14:00:14.559 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2022-01-11T14:00:14.559 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2022-01-11T14:00:14.559 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-01-11T14:00:14.560 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-01-11T14:00:14.560 INFO:teuthology.orchestra.run.smithi172.stdout: debug rgw = 20 2022-01-11T14:00:14.560 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt require ssl = False 2022-01-11T14:00:14.560 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt s3 kms backend = testing 2022-01-11T14:00:14.561 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2022-01-11T14:00:14.561 INFO:teuthology.orchestra.run.smithi172.stdout: rgw curl low speed time = 300 2022-01-11T14:00:14.561 INFO:teuthology.orchestra.run.smithi172.stdout: rgw data log num shards = 4 2022-01-11T14:00:14.561 INFO:teuthology.orchestra.run.smithi172.stdout: rgw md log max shards = 4 2022-01-11T14:00:14.561 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync log trim interval = 0 2022-01-11T14:00:14.562 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync meta inject err probability = 0.1 2022-01-11T14:00:14.562 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync obj etag verify = True 2022-01-11T14:00:14.562 INFO:teuthology.orchestra.run.smithi172.stdout: setgroup = ceph 2022-01-11T14:00:14.562 INFO:teuthology.orchestra.run.smithi172.stdout: setuser = ceph 2022-01-11T14:00:14.563 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2022-01-11T14:00:14.563 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2022-01-11T14:00:14.563 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 10 2022-01-11T14:00:14.563 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2022-01-11T14:00:14.563 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 1000 2022-01-11T14:00:14.564 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2022-01-11T14:00:14.564 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 10000 2022-01-11T14:00:14.565 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2022-01-11T14:00:14.566 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2022-01-11T14:00:14.656 INFO:tasks.ceph:Copying monmap to all nodes... 2022-01-11T14:00:14.656 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:14.657 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2022-01-11T14:00:14.671 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:14.672 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2022-01-11T14:00:14.725 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi161.front.sepia.ceph.com 2022-01-11T14:00:14.726 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T14:00:14.726 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/c2.keyring 2022-01-11T14:00:14.726 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-01-11T14:00:14.763 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-01-11T14:00:14.764 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/c2.monmap 2022-01-11T14:00:14.819 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2022-01-11T14:00:14.820 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:14.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/c2.keyring 2022-01-11T14:00:14.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/c2.keyring 2022-01-11T14:00:14.855 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:14.856 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/c2.monmap 2022-01-11T14:00:14.909 INFO:tasks.ceph:Setting up mon nodes... 2022-01-11T14:00:14.910 INFO:tasks.ceph:Setting up mgr nodes... 2022-01-11T14:00:14.911 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2022-01-11T14:00:14.997 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2022-01-11T14:00:14.999 INFO:tasks.ceph:Setting up mds nodes... 2022-01-11T14:00:14.999 INFO:tasks.ceph_client:Setting up client nodes... 2022-01-11T14:00:15.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2022-01-11T14:00:15.036 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/c2.client.0.keyring 2022-01-11T14:00:15.049 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2022-01-11T14:00:15.126 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/c2.client.1.keyring 2022-01-11T14:00:15.138 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-01-11T14:00:15.139 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2022-01-11T14:00:15.139 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2022-01-11T14:00:15.202 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-01-11T14:00:15.203 INFO:tasks.ceph:role: c2.osd.0 2022-01-11T14:00:15.203 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2022-01-11T14:00:15.203 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-01-11T14:00:15.273 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-01-11T14:00:15.274 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2022-01-11T14:00:15.274 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-01-11T14:00:15.274 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 2022-01-11T14:00:15.274 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-01-11T14:00:15.275 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2022-01-11T14:00:15.275 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-01-11T14:00:15.275 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-01-11T14:00:15.275 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-01-11T14:00:15.276 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-01-11T14:00:15.635 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2022-01-11T14:00:15.642 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2022-01-11T14:00:15.643 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2022-01-11T14:00:15.778 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2022-01-11T14:00:15.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2022-01-11T14:00:15.872 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-01-11T14:00:15.872 INFO:tasks.ceph:role: c2.osd.1 2022-01-11T14:00:15.872 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2022-01-11T14:00:15.873 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-01-11T14:00:15.943 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-01-11T14:00:15.944 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2022-01-11T14:00:15.944 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-01-11T14:00:15.944 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 2022-01-11T14:00:15.944 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-01-11T14:00:15.945 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2022-01-11T14:00:15.945 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-01-11T14:00:15.945 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-01-11T14:00:15.945 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-01-11T14:00:15.945 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-01-11T14:00:16.301 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2022-01-11T14:00:16.303 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2022-01-11T14:00:16.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2022-01-11T14:00:16.414 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2022-01-11T14:00:16.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2022-01-11T14:00:16.507 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2022-01-11T14:00:16.508 INFO:tasks.ceph:role: c2.osd.2 2022-01-11T14:00:16.508 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2022-01-11T14:00:16.509 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-01-11T14:00:16.579 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-01-11T14:00:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2022-01-11T14:00:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-01-11T14:00:16.580 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 2022-01-11T14:00:16.581 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-01-11T14:00:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2022-01-11T14:00:16.581 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-01-11T14:00:16.581 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-01-11T14:00:16.581 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-01-11T14:00:16.582 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-01-11T14:00:16.943 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2022-01-11T14:00:16.944 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2022-01-11T14:00:16.945 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2022-01-11T14:00:17.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2022-01-11T14:00:17.092 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2022-01-11T14:00:17.186 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:17.188+0000 7f27939e0240 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2022-01-11T14:00:17.187 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:17.189+0000 7f27939e0240 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2022-01-11T14:00:17.188 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:17.189+0000 7f27939e0240 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-01-11T14:00:17.188 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:17.189+0000 7f27939e0240 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-01-11T14:00:17.188 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:17.189+0000 7f27939e0240 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2022-01-11T14:00:17.189 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:17.190+0000 7f27939e0240 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2022-01-11T14:00:19.270 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2022-01-11T14:00:19.301 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2022-01-11T14:00:19.386 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:19.387+0000 7fec24855240 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2022-01-11T14:00:19.386 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:19.387+0000 7fec24855240 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2022-01-11T14:00:19.387 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:19.387+0000 7fec24855240 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-01-11T14:00:19.387 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:19.387+0000 7fec24855240 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-01-11T14:00:19.387 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:19.387+0000 7fec24855240 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2022-01-11T14:00:19.387 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:19.387+0000 7fec24855240 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2022-01-11T14:00:21.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2022-01-11T14:00:21.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2022-01-11T14:00:21.621 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:21.622+0000 7fb871f7f240 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2022-01-11T14:00:21.621 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:21.622+0000 7fb871f7f240 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2022-01-11T14:00:21.622 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:21.622+0000 7fb871f7f240 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-01-11T14:00:21.622 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:21.622+0000 7fb871f7f240 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-01-11T14:00:21.622 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:21.622+0000 7fb871f7f240 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2022-01-11T14:00:21.622 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:00:21.622+0000 7fb871f7f240 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2022-01-11T14:00:23.703 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2022-01-11T14:00:23.732 INFO:tasks.ceph:Reading keys from all nodes... 2022-01-11T14:00:23.733 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:23.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2022-01-11T14:00:23.798 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:23.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2022-01-11T14:00:23.864 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:23.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2022-01-11T14:00:23.930 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:23.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2022-01-11T14:00:23.997 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:23.998 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2022-01-11T14:00:24.052 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:24.053 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2022-01-11T14:00:24.108 INFO:tasks.ceph:Adding keys to all mons... 2022-01-11T14:00:24.108 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/c2.keyring 2022-01-11T14:00:24.171 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2022-01-11T14:00:24.171 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBvjd1hKCIDABAAxHBzvjD6IFHcjkp4gd0GOw== 2022-01-11T14:00:24.171 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2022-01-11T14:00:24.172 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBxjd1hIDZMCxAAMpltQsI0X5uIqyWu5Nstsg== 2022-01-11T14:00:24.172 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2022-01-11T14:00:24.172 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBzjd1h0ecoFxAA3Tp5SKSHLmyNRuFjkd8vDw== 2022-01-11T14:00:24.172 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2022-01-11T14:00:24.172 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQB1jd1hLoMsJRAAmipezW3OEBtmX42q3/38CA== 2022-01-11T14:00:24.173 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2022-01-11T14:00:24.173 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBvjd1hAShhAhAApbKw/jONSktmmDCmaEXgqg== 2022-01-11T14:00:24.173 INFO:teuthology.orchestra.run.smithi172.stdout:[client.1] 2022-01-11T14:00:24.173 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBvjd1hPcm3BxAAKh0yWIpZE7aqfXjKnxrXSw== 2022-01-11T14:00:24.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-01-11T14:00:24.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-01-11T14:00:24.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-01-11T14:00:24.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-01-11T14:00:24.403 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-01-11T14:00:24.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-01-11T14:00:24.501 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-01-11T14:00:24.503 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2022-01-11T14:00:24.527 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2022-01-11T14:00:24.618 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2022-01-11T14:00:24.642 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/c2.monmap 2022-01-11T14:00:24.697 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2022-01-11T14:00:24.698 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2022-01-11T14:00:24.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2022-01-11T14:00:24.739 INFO:tasks.ceph.c2.mon.a:Started 2022-01-11T14:00:24.740 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2022-01-11T14:00:24.740 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2022-01-11T14:00:24.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2022-01-11T14:00:24.743 INFO:tasks.ceph.c2.mgr.x:Started 2022-01-11T14:00:24.743 INFO:tasks.ceph:Setting crush tunables to default 2022-01-11T14:00:24.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd crush tunables default 2022-01-11T14:00:24.946 INFO:teuthology.orchestra.run.smithi172.stderr:adjusted tunables profile to default 2022-01-11T14:00:24.957 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2022-01-11T14:00:24.957 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:24.958 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2022-01-11T14:00:24.982 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:24.983 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2022-01-11T14:00:25.046 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2022-01-11T14:00:25.047 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2022-01-11T14:00:25.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd new d556c8bd-2d41-4fed-9ee5-ade02805cadc 0 2022-01-11T14:00:25.210 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:25.212+0000 7f9c644ef000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-01-11T14:00:25.345 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:25.346+0000 7f9c644ef000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-01-11T14:00:25.355 INFO:teuthology.orchestra.run.smithi172.stdout:0 2022-01-11T14:00:25.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd new 5fcb7d8d-e022-49eb-91b8-237e5b870fcc 1 2022-01-11T14:00:25.416 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:25.418+0000 7f9c644ef000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-01-11T14:00:25.572 INFO:teuthology.orchestra.run.smithi172.stdout:1 2022-01-11T14:00:25.584 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd new 789c9ec6-b5a8-41aa-9fa5-ababe695fe32 2 2022-01-11T14:00:25.788 INFO:teuthology.orchestra.run.smithi172.stdout:2 2022-01-11T14:00:25.799 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2022-01-11T14:00:25.800 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2022-01-11T14:00:25.802 INFO:tasks.ceph.c2.osd.0:Started 2022-01-11T14:00:25.803 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2022-01-11T14:00:25.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2022-01-11T14:00:25.805 INFO:tasks.ceph.c2.osd.1:Started 2022-01-11T14:00:25.805 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2022-01-11T14:00:25.806 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2022-01-11T14:00:25.808 INFO:tasks.ceph.c2.osd.2:Started 2022-01-11T14:00:25.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-01-11T14:00:26.006 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:26.006 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":5,"fsid":"65510500-97b1-4b61-b1ef-a0cc54b6a9d6","created":"2022-01-11T14:00:24.862394+0000","modified":"2022-01-11T14:00:25.789126+0000","last_up_change":"0.000000","last_in_change":"2022-01-11T14:00:25.789126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d556c8bd-2d41-4fed-9ee5-ade02805cadc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5fcb7d8d-e022-49eb-91b8-237e5b870fcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"789c9ec6-b5a8-41aa-9fa5-ababe695fe32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-11T14:00:26.017 INFO:tasks.ceph.ceph_manager.c2:[] 2022-01-11T14:00:26.018 INFO:tasks.ceph:Waiting for OSDs to come up 2022-01-11T14:00:26.187 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:26.188+0000 7f9c644ef000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-01-11T14:00:26.252 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:26.253+0000 7f9c644ef000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-01-11T14:00:26.321 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2022-01-11T14:00:26.325 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:26.326+0000 7f9c644ef000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-01-11T14:00:26.394 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:26.395+0000 7f9c644ef000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-01-11T14:00:26.400 INFO:tasks.ceph.c2.osd.2.smithi172.stderr:2022-01-11T14:00:26.401+0000 7f790bdcd240 -1 Falling back to public interface 2022-01-11T14:00:26.401 INFO:tasks.ceph.c2.osd.1.smithi172.stderr:2022-01-11T14:00:26.402+0000 7f510119f240 -1 Falling back to public interface 2022-01-11T14:00:26.410 INFO:tasks.ceph.c2.osd.0.smithi172.stderr:2022-01-11T14:00:26.411+0000 7fb1877aa240 -1 Falling back to public interface 2022-01-11T14:00:26.512 INFO:teuthology.misc.health.smithi172.stdout: 2022-01-11T14:00:26.512 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":5,"fsid":"65510500-97b1-4b61-b1ef-a0cc54b6a9d6","created":"2022-01-11T14:00:24.862394+0000","modified":"2022-01-11T14:00:25.789126+0000","last_up_change":"0.000000","last_in_change":"2022-01-11T14:00:25.789126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d556c8bd-2d41-4fed-9ee5-ade02805cadc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5fcb7d8d-e022-49eb-91b8-237e5b870fcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"789c9ec6-b5a8-41aa-9fa5-ababe695fe32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-11T14:00:26.523 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-01-11T14:00:26.539 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:26.541+0000 7f9c644ef000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-01-11T14:00:26.636 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:26.637+0000 7f9c644ef000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-01-11T14:00:26.964 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:26.966+0000 7f9c644ef000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-01-11T14:00:27.058 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:27.059+0000 7f9c644ef000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-01-11T14:00:27.132 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:27.133+0000 7f9c644ef000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-01-11T14:00:27.200 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:27.202+0000 7f9c644ef000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-01-11T14:00:27.471 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:27.472+0000 7f9c644ef000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-01-11T14:00:27.545 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:27.546+0000 7f9c644ef000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-01-11T14:00:27.682 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:27.684+0000 7f9c644ef000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-01-11T14:00:28.078 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:28.079+0000 7f9c644ef000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-01-11T14:00:28.182 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:28.183+0000 7f9c644ef000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-01-11T14:00:28.971 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:28.973+0000 7f9c644ef000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-01-11T14:00:29.042 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:29.044+0000 7f9c644ef000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-01-11T14:00:29.113 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:29.114+0000 7f9c644ef000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-01-11T14:00:29.197 INFO:tasks.ceph.c2.osd.1.smithi172.stderr:2022-01-11T14:00:29.198+0000 7f510119f240 -1 osd.1 0 log_to_monitors true 2022-01-11T14:00:29.211 INFO:tasks.ceph.c2.osd.2.smithi172.stderr:2022-01-11T14:00:29.213+0000 7f790bdcd240 -1 osd.2 0 log_to_monitors true 2022-01-11T14:00:29.211 INFO:tasks.ceph.c2.osd.0.smithi172.stderr:2022-01-11T14:00:29.213+0000 7fb1877aa240 -1 osd.0 0 log_to_monitors true 2022-01-11T14:00:29.769 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:29.771+0000 7f9c644ef000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-01-11T14:00:29.843 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:29.844+0000 7f9c644ef000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-01-11T14:00:30.027 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:00:30.028+0000 7f9c644ef000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-01-11T14:00:30.337 INFO:tasks.ceph.c2.osd.2.smithi172.stderr:2022-01-11T14:00:30.338+0000 7f79029ef700 -1 osd.2 0 waiting for initial osdmap 2022-01-11T14:00:30.339 INFO:tasks.ceph.c2.osd.1.smithi172.stderr:2022-01-11T14:00:30.340+0000 7f50f7dc1700 -1 osd.1 0 waiting for initial osdmap 2022-01-11T14:00:30.340 INFO:tasks.ceph.c2.osd.2.smithi172.stderr:2022-01-11T14:00:30.342+0000 7f78fb381700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-01-11T14:00:30.340 INFO:tasks.ceph.c2.osd.0.smithi172.stderr:2022-01-11T14:00:30.342+0000 7fb17e3cc700 -1 osd.0 0 waiting for initial osdmap 2022-01-11T14:00:30.341 INFO:tasks.ceph.c2.osd.1.smithi172.stderr:2022-01-11T14:00:30.343+0000 7f50f0753700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-01-11T14:00:30.342 INFO:tasks.ceph.c2.osd.0.smithi172.stderr:2022-01-11T14:00:30.344+0000 7fb176d5e700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-01-11T14:00:32.827 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2022-01-11T14:00:33.123 INFO:teuthology.misc.health.smithi172.stdout: 2022-01-11T14:00:33.124 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":9,"fsid":"65510500-97b1-4b61-b1ef-a0cc54b6a9d6","created":"2022-01-11T14:00:24.862394+0000","modified":"2022-01-11T14:00:33.053933+0000","last_up_change":"2022-01-11T14:00:31.341898+0000","last_in_change":"2022-01-11T14:00:25.789126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-01-11T14:00:32.090523+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"d556c8bd-2d41-4fed-9ee5-ade02805cadc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":32128},{"type":"v1","addr":"172.21.15.172:6817","nonce":32128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":32128},{"type":"v1","addr":"172.21.15.172:6819","nonce":32128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":32128},{"type":"v1","addr":"172.21.15.172:6823","nonce":32128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":32128},{"type":"v1","addr":"172.21.15.172:6821","nonce":32128}]},"public_addr":"172.21.15.172:6817/32128","cluster_addr":"172.21.15.172:6819/32128","heartbeat_back_addr":"172.21.15.172:6823/32128","heartbeat_front_addr":"172.21.15.172:6821/32128","state":["exists","up"]},{"osd":1,"uuid":"5fcb7d8d-e022-49eb-91b8-237e5b870fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":32129},{"type":"v1","addr":"172.21.15.172:6809","nonce":32129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":32129},{"type":"v1","addr":"172.21.15.172:6811","nonce":32129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":32129},{"type":"v1","addr":"172.21.15.172:6815","nonce":32129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":32129},{"type":"v1","addr":"172.21.15.172:6813","nonce":32129}]},"public_addr":"172.21.15.172:6809/32129","cluster_addr":"172.21.15.172:6811/32129","heartbeat_back_addr":"172.21.15.172:6815/32129","heartbeat_front_addr":"172.21.15.172:6813/32129","state":["exists","up"]},{"osd":2,"uuid":"789c9ec6-b5a8-41aa-9fa5-ababe695fe32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":32130},{"type":"v1","addr":"172.21.15.172:6801","nonce":32130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":32130},{"type":"v1","addr":"172.21.15.172:6803","nonce":32130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":32130},{"type":"v1","addr":"172.21.15.172:6807","nonce":32130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":32130},{"type":"v1","addr":"172.21.15.172:6805","nonce":32130}]},"public_addr":"172.21.15.172:6801/32130","cluster_addr":"172.21.15.172:6803/32130","heartbeat_back_addr":"172.21.15.172:6807/32130","heartbeat_front_addr":"172.21.15.172:6805/32130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:30.231840+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:30.243943+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:30.180062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-11T14:00:33.136 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-01-11T14:00:33.136 INFO:tasks.ceph:Creating RBD pool 2022-01-11T14:00:33.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd pool create rbd 8 2022-01-11T14:00:34.065 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'rbd' created 2022-01-11T14:00:34.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2022-01-11T14:00:35.063 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rbd' on pool 'rbd' 2022-01-11T14:00:35.077 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2022-01-11T14:00:35.078 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2022-01-11T14:00:35.078 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2022-01-11T14:00:35.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2022-01-11T14:00:35.079 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-01-11T14:00:35.423 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:35.434 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":31791},{"type":"v1","addr":"172.21.15.172:6825","nonce":31791}]},"active_addr":"172.21.15.172:6825/31791","active_change":"2022-01-11T14:00:30.043317+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":1234241582}]},{"addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":1618787430}]},{"addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":2837574631}]}]}} 2022-01-11T14:00:35.437 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2022-01-11T14:00:35.438 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2022-01-11T14:00:35.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-01-11T14:00:35.733 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:35.733 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":11,"fsid":"65510500-97b1-4b61-b1ef-a0cc54b6a9d6","created":"2022-01-11T14:00:24.862394+0000","modified":"2022-01-11T14:00:35.059351+0000","last_up_change":"2022-01-11T14:00:31.341898+0000","last_in_change":"2022-01-11T14:00:25.789126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-01-11T14:00:32.090523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-01-11T14:00:33.449550+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d556c8bd-2d41-4fed-9ee5-ade02805cadc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":32128},{"type":"v1","addr":"172.21.15.172:6817","nonce":32128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":32128},{"type":"v1","addr":"172.21.15.172:6819","nonce":32128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":32128},{"type":"v1","addr":"172.21.15.172:6823","nonce":32128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":32128},{"type":"v1","addr":"172.21.15.172:6821","nonce":32128}]},"public_addr":"172.21.15.172:6817/32128","cluster_addr":"172.21.15.172:6819/32128","heartbeat_back_addr":"172.21.15.172:6823/32128","heartbeat_front_addr":"172.21.15.172:6821/32128","state":["exists","up"]},{"osd":1,"uuid":"5fcb7d8d-e022-49eb-91b8-237e5b870fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":32129},{"type":"v1","addr":"172.21.15.172:6809","nonce":32129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":32129},{"type":"v1","addr":"172.21.15.172:6811","nonce":32129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":32129},{"type":"v1","addr":"172.21.15.172:6815","nonce":32129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":32129},{"type":"v1","addr":"172.21.15.172:6813","nonce":32129}]},"public_addr":"172.21.15.172:6809/32129","cluster_addr":"172.21.15.172:6811/32129","heartbeat_back_addr":"172.21.15.172:6815/32129","heartbeat_front_addr":"172.21.15.172:6813/32129","state":["exists","up"]},{"osd":2,"uuid":"789c9ec6-b5a8-41aa-9fa5-ababe695fe32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":32130},{"type":"v1","addr":"172.21.15.172:6801","nonce":32130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":32130},{"type":"v1","addr":"172.21.15.172:6803","nonce":32130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":32130},{"type":"v1","addr":"172.21.15.172:6807","nonce":32130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":32130},{"type":"v1","addr":"172.21.15.172:6805","nonce":32130}]},"public_addr":"172.21.15.172:6801/32130","cluster_addr":"172.21.15.172:6803/32130","heartbeat_back_addr":"172.21.15.172:6807/32130","heartbeat_front_addr":"172.21.15.172:6805/32130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:30.231840+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:30.243943+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:30.180062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-11T14:00:35.746 INFO:tasks.ceph.ceph_manager.c2:all up! 2022-01-11T14:00:35.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2022-01-11T14:00:36.040 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:36.040 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":11,"fsid":"65510500-97b1-4b61-b1ef-a0cc54b6a9d6","created":"2022-01-11T14:00:24.862394+0000","modified":"2022-01-11T14:00:35.059351+0000","last_up_change":"2022-01-11T14:00:31.341898+0000","last_in_change":"2022-01-11T14:00:25.789126+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-01-11T14:00:32.090523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-01-11T14:00:33.449550+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d556c8bd-2d41-4fed-9ee5-ade02805cadc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":32128},{"type":"v1","addr":"172.21.15.172:6817","nonce":32128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":32128},{"type":"v1","addr":"172.21.15.172:6819","nonce":32128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":32128},{"type":"v1","addr":"172.21.15.172:6823","nonce":32128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":32128},{"type":"v1","addr":"172.21.15.172:6821","nonce":32128}]},"public_addr":"172.21.15.172:6817/32128","cluster_addr":"172.21.15.172:6819/32128","heartbeat_back_addr":"172.21.15.172:6823/32128","heartbeat_front_addr":"172.21.15.172:6821/32128","state":["exists","up"]},{"osd":1,"uuid":"5fcb7d8d-e022-49eb-91b8-237e5b870fcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":32129},{"type":"v1","addr":"172.21.15.172:6809","nonce":32129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":32129},{"type":"v1","addr":"172.21.15.172:6811","nonce":32129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":32129},{"type":"v1","addr":"172.21.15.172:6815","nonce":32129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":32129},{"type":"v1","addr":"172.21.15.172:6813","nonce":32129}]},"public_addr":"172.21.15.172:6809/32129","cluster_addr":"172.21.15.172:6811/32129","heartbeat_back_addr":"172.21.15.172:6815/32129","heartbeat_front_addr":"172.21.15.172:6813/32129","state":["exists","up"]},{"osd":2,"uuid":"789c9ec6-b5a8-41aa-9fa5-ababe695fe32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":32130},{"type":"v1","addr":"172.21.15.172:6801","nonce":32130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":32130},{"type":"v1","addr":"172.21.15.172:6803","nonce":32130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":32130},{"type":"v1","addr":"172.21.15.172:6807","nonce":32130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":32130},{"type":"v1","addr":"172.21.15.172:6805","nonce":32130}]},"public_addr":"172.21.15.172:6801/32130","cluster_addr":"172.21.15.172:6803/32130","heartbeat_back_addr":"172.21.15.172:6807/32130","heartbeat_front_addr":"172.21.15.172:6805/32130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:30.231840+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:30.243943+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-01-11T14:00:30.180062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-01-11T14:00:36.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2022-01-11T14:00:36.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2022-01-11T14:00:36.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2022-01-11T14:00:36.209 INFO:teuthology.orchestra.run.smithi172.stdout:34359738371 2022-01-11T14:00:36.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-01-11T14:00:36.237 INFO:teuthology.orchestra.run.smithi172.stdout:34359738371 2022-01-11T14:00:36.238 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-01-11T14:00:36.239 INFO:teuthology.orchestra.run.smithi172.stdout:34359738371 2022-01-11T14:00:36.239 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-01-11T14:00:36.529 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2022-01-11T14:00:36.543 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2022-01-11T14:00:36.589 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2022-01-11T14:00:36.596 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2022-01-11T14:00:36.604 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.1 2022-01-11T14:00:36.611 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.2 2022-01-11T14:00:37.544 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-01-11T14:00:37.605 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-01-11T14:00:37.614 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-01-11T14:00:37.870 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2022-01-11T14:00:37.884 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2022-01-11T14:00:37.921 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2022-01-11T14:00:37.935 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.1 2022-01-11T14:00:37.958 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2022-01-11T14:00:37.972 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.2 2022-01-11T14:00:38.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2022-01-11T14:00:38.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2022-01-11T14:00:38.973 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2022-01-11T14:00:39.229 INFO:teuthology.orchestra.run.smithi172.stdout:34359738372 2022-01-11T14:00:39.242 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738372 for osd.0 2022-01-11T14:00:39.243 DEBUG:teuthology.parallel:result is None 2022-01-11T14:00:39.267 INFO:teuthology.orchestra.run.smithi172.stdout:34359738372 2022-01-11T14:00:39.280 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738372 for osd.1 2022-01-11T14:00:39.281 DEBUG:teuthology.parallel:result is None 2022-01-11T14:00:39.292 INFO:teuthology.orchestra.run.smithi172.stdout:34359738372 2022-01-11T14:00:39.305 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738372 for osd.2 2022-01-11T14:00:39.306 DEBUG:teuthology.parallel:result is None 2022-01-11T14:00:39.306 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2022-01-11T14:00:39.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2022-01-11T14:00:39.632 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:39.634 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2022-01-11T14:00:39.645 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2022-01-11T14:00:38.050338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":17568,"kb_used_data":1224,"kb_used_omap":0,"kb_used_meta":16320,"kb_avail":314555232,"statfs":{"total":322122547200,"available":322104557568,"internally_reserved":0,"allocated":1253376,"data_stored":1017659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16711680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.068656+0000","last_change":"2022-01-11T14:00:35.068656+0000","last_active":"2022-01-11T14:00:35.068656+0000","last_peered":"2022-01-11T14:00:35.068656+0000","last_clean":"2022-01-11T14:00:35.068656+0000","last_became_active":"2022-01-11T14:00:35.068295+0000","last_became_peered":"2022-01-11T14:00:35.068295+0000","last_unstale":"2022-01-11T14:00:35.068656+0000","last_undegraded":"2022-01-11T14:00:35.068656+0000","last_fullsized":"2022-01-11T14:00:35.068656+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T21:36:15.509646+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.065740+0000","last_change":"2022-01-11T14:00:35.065740+0000","last_active":"2022-01-11T14:00:35.065740+0000","last_peered":"2022-01-11T14:00:35.065740+0000","last_clean":"2022-01-11T14:00:35.065740+0000","last_became_active":"2022-01-11T14:00:35.065473+0000","last_became_peered":"2022-01-11T14:00:35.065473+0000","last_unstale":"2022-01-11T14:00:35.065740+0000","last_undegraded":"2022-01-11T14:00:35.065740+0000","last_fullsized":"2022-01-11T14:00:35.065740+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-13T00:02:57.716845+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.068425+0000","last_change":"2022-01-11T14:00:35.068425+0000","last_active":"2022-01-11T14:00:35.068425+0000","last_peered":"2022-01-11T14:00:35.068425+0000","last_clean":"2022-01-11T14:00:35.068425+0000","last_became_active":"2022-01-11T14:00:35.068044+0000","last_became_peered":"2022-01-11T14:00:35.068044+0000","last_unstale":"2022-01-11T14:00:35.068425+0000","last_undegraded":"2022-01-11T14:00:35.068425+0000","last_fullsized":"2022-01-11T14:00:35.068425+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-13T00:02:01.949053+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.069475+0000","last_change":"2022-01-11T14:00:35.068129+0000","last_active":"2022-01-11T14:00:35.069475+0000","last_peered":"2022-01-11T14:00:35.069475+0000","last_clean":"2022-01-11T14:00:35.069475+0000","last_became_active":"2022-01-11T14:00:35.067864+0000","last_became_peered":"2022-01-11T14:00:35.067864+0000","last_unstale":"2022-01-11T14:00:35.069475+0000","last_undegraded":"2022-01-11T14:00:35.069475+0000","last_fullsized":"2022-01-11T14:00:35.069475+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T15:47:05.903775+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2022-01-11T14:00:34.495134+0000","last_change":"2022-01-11T14:00:34.071912+0000","last_active":"2022-01-11T14:00:34.495134+0000","last_peered":"2022-01-11T14:00:34.495134+0000","last_clean":"2022-01-11T14:00:34.495134+0000","last_became_active":"2022-01-11T14:00:34.070875+0000","last_became_peered":"2022-01-11T14:00:34.070875+0000","last_unstale":"2022-01-11T14:00:34.495134+0000","last_undegraded":"2022-01-11T14:00:34.495134+0000","last_fullsized":"2022-01-11T14:00:34.495134+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:33.053933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:33.053933+0000","last_clean_scrub_stamp":"2022-01-11T14:00:33.053933+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T21:35:12.048932+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.064423+0000","last_change":"2022-01-11T14:00:35.064423+0000","last_active":"2022-01-11T14:00:35.064423+0000","last_peered":"2022-01-11T14:00:35.064423+0000","last_clean":"2022-01-11T14:00:35.064423+0000","last_became_active":"2022-01-11T14:00:35.064150+0000","last_became_peered":"2022-01-11T14:00:35.064150+0000","last_unstale":"2022-01-11T14:00:35.064423+0000","last_undegraded":"2022-01-11T14:00:35.064423+0000","last_fullsized":"2022-01-11T14:00:35.064423+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T15:54:50.804955+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.070139+0000","last_change":"2022-01-11T14:00:35.069316+0000","last_active":"2022-01-11T14:00:35.070139+0000","last_peered":"2022-01-11T14:00:35.070139+0000","last_clean":"2022-01-11T14:00:35.070139+0000","last_became_active":"2022-01-11T14:00:35.069004+0000","last_became_peered":"2022-01-11T14:00:35.069004+0000","last_unstale":"2022-01-11T14:00:35.070139+0000","last_undegraded":"2022-01-11T14:00:35.070139+0000","last_fullsized":"2022-01-11T14:00:35.070139+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-13T01:17:42.034407+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.070131+0000","last_change":"2022-01-11T14:00:35.068150+0000","last_active":"2022-01-11T14:00:35.070131+0000","last_peered":"2022-01-11T14:00:35.070131+0000","last_clean":"2022-01-11T14:00:35.070131+0000","last_became_active":"2022-01-11T14:00:35.067828+0000","last_became_peered":"2022-01-11T14:00:35.067828+0000","last_unstale":"2022-01-11T14:00:35.070131+0000","last_undegraded":"2022-01-11T14:00:35.070131+0000","last_fullsized":"2022-01-11T14:00:35.070131+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-13T01:16:10.683162+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.065106+0000","last_change":"2022-01-11T14:00:35.065106+0000","last_active":"2022-01-11T14:00:35.065106+0000","last_peered":"2022-01-11T14:00:35.065106+0000","last_clean":"2022-01-11T14:00:35.065106+0000","last_became_active":"2022-01-11T14:00:35.064840+0000","last_became_peered":"2022-01-11T14:00:35.064840+0000","last_unstale":"2022-01-11T14:00:35.065106+0000","last_undegraded":"2022-01-11T14:00:35.065106+0000","last_fullsized":"2022-01-11T14:00:35.065106+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T17:29:42.211424+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5296,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":104852304,"statfs":{"total":107374182400,"available":107368759296,"internally_reserved":0,"allocated":106496,"data_stored":29071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":6140,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":104851460,"statfs":{"total":107374182400,"available":107367895040,"internally_reserved":0,"allocated":577536,"data_stored":496027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":6132,"kb_used_data":556,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":104851468,"statfs":{"total":107374182400,"available":107367903232,"internally_reserved":0,"allocated":569344,"data_stored":492561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-01-11T14:00:39.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2022-01-11T14:00:39.934 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:39.936 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2022-01-11T14:00:39.947 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2022-01-11T14:00:38.050338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":17568,"kb_used_data":1224,"kb_used_omap":0,"kb_used_meta":16320,"kb_avail":314555232,"statfs":{"total":322122547200,"available":322104557568,"internally_reserved":0,"allocated":1253376,"data_stored":1017659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16711680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.068656+0000","last_change":"2022-01-11T14:00:35.068656+0000","last_active":"2022-01-11T14:00:35.068656+0000","last_peered":"2022-01-11T14:00:35.068656+0000","last_clean":"2022-01-11T14:00:35.068656+0000","last_became_active":"2022-01-11T14:00:35.068295+0000","last_became_peered":"2022-01-11T14:00:35.068295+0000","last_unstale":"2022-01-11T14:00:35.068656+0000","last_undegraded":"2022-01-11T14:00:35.068656+0000","last_fullsized":"2022-01-11T14:00:35.068656+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T21:36:15.509646+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.065740+0000","last_change":"2022-01-11T14:00:35.065740+0000","last_active":"2022-01-11T14:00:35.065740+0000","last_peered":"2022-01-11T14:00:35.065740+0000","last_clean":"2022-01-11T14:00:35.065740+0000","last_became_active":"2022-01-11T14:00:35.065473+0000","last_became_peered":"2022-01-11T14:00:35.065473+0000","last_unstale":"2022-01-11T14:00:35.065740+0000","last_undegraded":"2022-01-11T14:00:35.065740+0000","last_fullsized":"2022-01-11T14:00:35.065740+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-13T00:02:57.716845+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.068425+0000","last_change":"2022-01-11T14:00:35.068425+0000","last_active":"2022-01-11T14:00:35.068425+0000","last_peered":"2022-01-11T14:00:35.068425+0000","last_clean":"2022-01-11T14:00:35.068425+0000","last_became_active":"2022-01-11T14:00:35.068044+0000","last_became_peered":"2022-01-11T14:00:35.068044+0000","last_unstale":"2022-01-11T14:00:35.068425+0000","last_undegraded":"2022-01-11T14:00:35.068425+0000","last_fullsized":"2022-01-11T14:00:35.068425+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-13T00:02:01.949053+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.069475+0000","last_change":"2022-01-11T14:00:35.068129+0000","last_active":"2022-01-11T14:00:35.069475+0000","last_peered":"2022-01-11T14:00:35.069475+0000","last_clean":"2022-01-11T14:00:35.069475+0000","last_became_active":"2022-01-11T14:00:35.067864+0000","last_became_peered":"2022-01-11T14:00:35.067864+0000","last_unstale":"2022-01-11T14:00:35.069475+0000","last_undegraded":"2022-01-11T14:00:35.069475+0000","last_fullsized":"2022-01-11T14:00:35.069475+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T15:47:05.903775+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2022-01-11T14:00:34.495134+0000","last_change":"2022-01-11T14:00:34.071912+0000","last_active":"2022-01-11T14:00:34.495134+0000","last_peered":"2022-01-11T14:00:34.495134+0000","last_clean":"2022-01-11T14:00:34.495134+0000","last_became_active":"2022-01-11T14:00:34.070875+0000","last_became_peered":"2022-01-11T14:00:34.070875+0000","last_unstale":"2022-01-11T14:00:34.495134+0000","last_undegraded":"2022-01-11T14:00:34.495134+0000","last_fullsized":"2022-01-11T14:00:34.495134+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:33.053933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:33.053933+0000","last_clean_scrub_stamp":"2022-01-11T14:00:33.053933+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T21:35:12.048932+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.064423+0000","last_change":"2022-01-11T14:00:35.064423+0000","last_active":"2022-01-11T14:00:35.064423+0000","last_peered":"2022-01-11T14:00:35.064423+0000","last_clean":"2022-01-11T14:00:35.064423+0000","last_became_active":"2022-01-11T14:00:35.064150+0000","last_became_peered":"2022-01-11T14:00:35.064150+0000","last_unstale":"2022-01-11T14:00:35.064423+0000","last_undegraded":"2022-01-11T14:00:35.064423+0000","last_fullsized":"2022-01-11T14:00:35.064423+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T15:54:50.804955+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.070139+0000","last_change":"2022-01-11T14:00:35.069316+0000","last_active":"2022-01-11T14:00:35.070139+0000","last_peered":"2022-01-11T14:00:35.070139+0000","last_clean":"2022-01-11T14:00:35.070139+0000","last_became_active":"2022-01-11T14:00:35.069004+0000","last_became_peered":"2022-01-11T14:00:35.069004+0000","last_unstale":"2022-01-11T14:00:35.070139+0000","last_undegraded":"2022-01-11T14:00:35.070139+0000","last_fullsized":"2022-01-11T14:00:35.070139+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-13T01:17:42.034407+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.070131+0000","last_change":"2022-01-11T14:00:35.068150+0000","last_active":"2022-01-11T14:00:35.070131+0000","last_peered":"2022-01-11T14:00:35.070131+0000","last_clean":"2022-01-11T14:00:35.070131+0000","last_became_active":"2022-01-11T14:00:35.067828+0000","last_became_peered":"2022-01-11T14:00:35.067828+0000","last_unstale":"2022-01-11T14:00:35.070131+0000","last_undegraded":"2022-01-11T14:00:35.070131+0000","last_fullsized":"2022-01-11T14:00:35.070131+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-13T01:16:10.683162+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2022-01-11T14:00:35.065106+0000","last_change":"2022-01-11T14:00:35.065106+0000","last_active":"2022-01-11T14:00:35.065106+0000","last_peered":"2022-01-11T14:00:35.065106+0000","last_clean":"2022-01-11T14:00:35.065106+0000","last_became_active":"2022-01-11T14:00:35.064840+0000","last_became_peered":"2022-01-11T14:00:35.064840+0000","last_unstale":"2022-01-11T14:00:35.065106+0000","last_undegraded":"2022-01-11T14:00:35.065106+0000","last_fullsized":"2022-01-11T14:00:35.065106+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-01-11T14:00:34.058558+0000","last_clean_scrub_stamp":"2022-01-11T14:00:34.058558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-01-12T17:29:42.211424+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5296,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":104852304,"statfs":{"total":107374182400,"available":107368759296,"internally_reserved":0,"allocated":106496,"data_stored":29071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":6140,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":104851460,"statfs":{"total":107374182400,"available":107367895040,"internally_reserved":0,"allocated":577536,"data_stored":496027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":6132,"kb_used_data":556,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":104851468,"statfs":{"total":107374182400,"available":107367903232,"internally_reserved":0,"allocated":569344,"data_stored":492561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-01-11T14:00:39.948 INFO:tasks.ceph.ceph_manager.c2:clean! 2022-01-11T14:00:39.948 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2022-01-11T14:00:39.948 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2022-01-11T14:00:39.949 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2022-01-11T14:00:40.276 INFO:teuthology.orchestra.run.smithi172.stdout: 2022-01-11T14:00:40.276 INFO:teuthology.orchestra.run.smithi172.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-01-11T14:00:40.289 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2022-01-11T14:00:40.290 INFO:teuthology.run_tasks:Running task rgw... 2022-01-11T14:00:40.305 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2022-01-11T14:00:40.305 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2022-01-11T14:00:40.306 INFO:tasks.rgw:Creating data pools 2022-01-11T14:00:40.306 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2022-01-11T14:00:40.307 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2022-01-11T14:00:40.681 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'default.rgw.buckets.data' created 2022-01-11T14:00:40.696 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2022-01-11T14:00:41.704 INFO:teuthology.orchestra.run.smithi161.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-01-11T14:00:41.715 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2022-01-11T14:00:42.704 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'default.rgw.buckets.index' created 2022-01-11T14:00:42.714 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2022-01-11T14:00:43.709 INFO:teuthology.orchestra.run.smithi161.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-01-11T14:00:43.729 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2022-01-11T14:00:43.729 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2022-01-11T14:00:44.021 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'default.rgw.buckets.data' already exists 2022-01-11T14:00:44.031 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2022-01-11T14:00:44.705 INFO:teuthology.orchestra.run.smithi161.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-01-11T14:00:44.717 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2022-01-11T14:00:45.009 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'default.rgw.buckets.index' already exists 2022-01-11T14:00:45.020 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2022-01-11T14:00:45.713 INFO:teuthology.orchestra.run.smithi161.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-01-11T14:00:45.724 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2022-01-11T14:00:45.725 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2022-01-11T14:00:46.847 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'default.rgw.buckets.data' created 2022-01-11T14:00:46.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2022-01-11T14:00:47.855 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-01-11T14:00:47.872 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2022-01-11T14:00:48.871 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'default.rgw.buckets.index' created 2022-01-11T14:00:48.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2022-01-11T14:00:49.873 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-01-11T14:00:49.893 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2022-01-11T14:00:49.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2022-01-11T14:00:50.190 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'default.rgw.buckets.data' already exists 2022-01-11T14:00:50.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2022-01-11T14:00:50.871 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2022-01-11T14:00:50.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2022-01-11T14:00:51.174 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'default.rgw.buckets.index' already exists 2022-01-11T14:00:51.185 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2022-01-11T14:00:51.872 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2022-01-11T14:00:51.886 DEBUG:tasks.rgw:Pools created 2022-01-11T14:00:51.886 INFO:tasks.rgw:Configuring compression type = random 2022-01-11T14:00:51.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2022-01-11T14:00:51.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2022-01-11T14:00:51.888 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2022-01-11T14:00:51.998 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:00:51.999 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:00:52.005 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:52.007+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.455 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.456+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:53.456 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.456+0000 7fdadf26b500 20 realm 2022-01-11T14:00:53.456 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.456+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.457 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.458+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:53.457 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.458+0000 7fdadf26b500 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-01-11T14:00:53.457 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.458+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.458 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.459+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:53.458 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.459+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.459 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.460+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:53.459 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.460+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.460+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.460+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.460+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.460+0000 7fdadf26b500 10 Creating default zonegroup 2022-01-11T14:00:53.460 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.460+0000 7fdadf26b500 20 get_system_obj_state: rctx=0x7fff4450a770 obj=:.pools.avail state=0x55b74712ec10 s->prefetch_data=0 2022-01-11T14:00:53.461 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.461+0000 7fdadf26b500 10 couldn't find old data placement pools config, setting up new ones for the zone 2022-01-11T14:00:53.468 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.470+0000 7fdadf26b500 10 WARNING: store->list_zones() returned r=-2 2022-01-11T14:00:53.469 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.470+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.469 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.470+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:53.471 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.473+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.472 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.473+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:53.472 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.473+0000 7fdadf26b500 10 could not read realm id: (2) No such file or directory 2022-01-11T14:00:53.472 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.473+0000 7fdadf26b500 10 WARNING: failed to set zone as default, r=-22 2022-01-11T14:00:53.472 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.473+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.473 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.473+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:53.476 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.478+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.479+0000 7fdadf26b500 20 rados_obj.operate() r=0 bl.length=873 2022-01-11T14:00:53.477 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.479+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.479+0000 7fdadf26b500 20 rados_obj.operate() r=0 bl.length=362 2022-01-11T14:00:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.479+0000 7fdadf26b500 10 Using default name default 2022-01-11T14:00:53.478 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.479+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.479 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.480+0000 7fdadf26b500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:00:53.480 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.480+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.480 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.480+0000 7fdadf26b500 20 rados_obj.operate() r=0 bl.length=873 2022-01-11T14:00:53.480 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.480+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.480 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.480+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:53.481 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.480+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.481 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.480+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:53.481 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.480+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.482 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.484+0000 7fdadf26b500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:00:53.490 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.491+0000 7fdadf26b500 20 RGWRados::pool_iterate: got zonegroup_info.6e25cdb6-f53f-43b0-83cd-3db14c73dfeb 2022-01-11T14:00:53.490 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.491+0000 7fdadf26b500 20 RGWRados::pool_iterate: got zone_info.13be4714-7bb8-4369-8a42-f7f76c38c08c 2022-01-11T14:00:53.490 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.491+0000 7fdadf26b500 20 RGWRados::pool_iterate: got zone_names.default 2022-01-11T14:00:53.491 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.491+0000 7fdadf26b500 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-01-11T14:00:53.491 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.491+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.491 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.492+0000 7fdadf26b500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:00:53.491 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.492+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.492 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.492+0000 7fdadf26b500 20 rados_obj.operate() r=0 bl.length=873 2022-01-11T14:00:53.492 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.492+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.493 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.492+0000 7fdadf26b500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:00:53.494 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.492+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.494 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.492+0000 7fdadf26b500 20 rados_obj.operate() r=0 bl.length=362 2022-01-11T14:00:53.494 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.492+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.494 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.493+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:53.495 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.493+0000 7fdadf26b500 20 zone default found 2022-01-11T14:00:53.495 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.493+0000 7fdadf26b500 4 Realm: () 2022-01-11T14:00:53.495 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.493+0000 7fdadf26b500 4 ZoneGroup: default (6e25cdb6-f53f-43b0-83cd-3db14c73dfeb) 2022-01-11T14:00:53.495 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.493+0000 7fdadf26b500 4 Zone: default (13be4714-7bb8-4369-8a42-f7f76c38c08c) 2022-01-11T14:00:53.495 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.493+0000 7fdadf26b500 10 cannot find current period zonegroup using local zonegroup configuration 2022-01-11T14:00:53.496 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.493+0000 7fdadf26b500 20 zonegroup default 2022-01-11T14:00:53.496 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.493+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:53.496 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.493+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:53.496 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:53.493+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:55.465 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:55.466+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:55.465 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:55.466+0000 7fdadf26b500 20 rados->read ofs=0 len=0 2022-01-11T14:00:55.466 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:55.467+0000 7fdadf26b500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:00:55.466 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:55.467+0000 7fdadf26b500 20 started sync module instance, tier type = 2022-01-11T14:00:55.466 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:55.467+0000 7fdadf26b500 20 started zone id=13be4714-7bb8-4369-8a42-f7f76c38c08c (name=default) with tier type = 2022-01-11T14:00:57.496 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.497+0000 7fdadf26b500 20 add_watcher() i=0 2022-01-11T14:00:57.496 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.497+0000 7fdadf26b500 20 add_watcher() i=1 2022-01-11T14:00:57.496 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.497+0000 7fdadf26b500 20 add_watcher() i=2 2022-01-11T14:00:57.497 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.497+0000 7fdadf26b500 20 add_watcher() i=3 2022-01-11T14:00:57.497 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.497+0000 7fdadf26b500 20 add_watcher() i=4 2022-01-11T14:00:57.497 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.497+0000 7fdadf26b500 20 add_watcher() i=5 2022-01-11T14:00:57.497 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.497+0000 7fdadf26b500 20 add_watcher() i=6 2022-01-11T14:00:57.499 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.498+0000 7fdadf26b500 20 add_watcher() i=7 2022-01-11T14:00:57.499 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.498+0000 7fdadf26b500 2 all 8 watchers are set, enabling cache 2022-01-11T14:00:57.499 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.499+0000 7fdadf26b500 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2022-01-11T14:00:57.500 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.499+0000 7fdadf26b500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2022-01-11T14:00:57.500 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.499+0000 7fdadf26b500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-01-11T14:00:57.500 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.499+0000 7fdadf26b500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2022-01-11T14:00:57.500 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.499+0000 7fdadf26b500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2022-01-11T14:00:57.501 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.499+0000 7fdadf26b500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-01-11T14:00:57.501 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.499+0000 7fdadf26b500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2022-01-11T14:00:57.501 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.499+0000 7fdadf26b500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2022-01-11T14:00:57.501 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.499+0000 7fdadf26b500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-01-11T14:00:57.501 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.499+0000 7fdadf26b500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2022-01-11T14:00:57.502 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.499+0000 7fdadf26b500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2022-01-11T14:00:57.502 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.499+0000 7fdadf26b500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-01-11T14:00:57.502 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.500+0000 7fdadf26b500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2022-01-11T14:00:57.502 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.500+0000 7fdadf26b500 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2022-01-11T14:00:57.503 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.501+0000 7fdadf26b500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-01-11T14:00:57.503 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.504+0000 7fdaa3fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-01-11T14:00:57.504 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.504+0000 7fdadf26b500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-01-11T14:00:57.504 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:00:57.504+0000 7fdadf26b500 20 check_secure_mon_conn(): mode 1 is insecure 2022-01-11T14:01:01.469 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.471+0000 7fdadf26b500 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2022-01-11T14:01:01.470 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.471+0000 7fdadf26b500 5 note: GC not initialized 2022-01-11T14:01:01.471 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.473+0000 7fda2affd700 20 reqs_thread_entry: start 2022-01-11T14:01:01.472 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.473+0000 7fda28ff9700 20 reqs_thread_entry: start 2022-01-11T14:01:01.472 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.473+0000 7fdadf26b500 20 init_complete bucket index max shards: 11 2022-01-11T14:01:01.472 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.473+0000 7fdadf26b500 10 rgw notify: Started notification manager with: 1 workers 2022-01-11T14:01:01.472 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.473+0000 7fd9faffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 11 14:01:31 2022 2022-01-11T14:01:01.473 INFO:teuthology.orchestra.run.smithi161.stderr: 2022-01-11T14:01:01.473 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.473+0000 7fd9fa7fc700 20 reqs_thread_entry: start 2022-01-11T14:01:01.494 INFO:teuthology.orchestra.run.smithi161.stdout:[] 2022-01-11T14:01:01.495 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.496+0000 7fdadf26b500 20 remove_watcher() i=0 2022-01-11T14:01:01.495 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.496+0000 7fdadf26b500 2 removed watcher, disabling cache 2022-01-11T14:01:01.496 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.497+0000 7fdadf26b500 20 remove_watcher() i=1 2022-01-11T14:01:01.498 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.500+0000 7fdadf26b500 20 remove_watcher() i=2 2022-01-11T14:01:01.501 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.503+0000 7fdadf26b500 20 remove_watcher() i=3 2022-01-11T14:01:01.503 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.505+0000 7fdadf26b500 20 remove_watcher() i=4 2022-01-11T14:01:01.506 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.508+0000 7fdadf26b500 20 remove_watcher() i=5 2022-01-11T14:01:01.507 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.509+0000 7fdadf26b500 20 remove_watcher() i=6 2022-01-11T14:01:01.508 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.510+0000 7fdadf26b500 20 remove_watcher() i=7 2022-01-11T14:01:01.509 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.511+0000 7fdaa3fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-01-11T14:01:01.515 DEBUG:tasks.util.rgw: json result: [] 2022-01-11T14:01:01.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-01-11T14:01:01.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-01-11T14:01:01.516 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-01-11T14:01:01.562 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:01.563 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.570+0000 7fa7a6a72500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-01-11T14:01:01.568 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.570+0000 7fa7a6a72500 20 check_secure_mon_conn(): mode 1 is insecure 2022-01-11T14:01:01.571 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.573+0000 7fa6e2ffd700 20 reqs_thread_entry: start 2022-01-11T14:01:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.573+0000 7fa7a6a72500 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-01-11T14:01:01.572 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.573+0000 7fa7a6a72500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.574+0000 7fa7a6a72500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.574+0000 7fa7a6a72500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.573 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.575+0000 7fa7a6a72500 20 rados_obj.operate() r=0 bl.length=873 2022-01-11T14:01:01.581 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.582+0000 7fa7a6a72500 20 RGWRados::pool_iterate: got zonegroup_info.6e25cdb6-f53f-43b0-83cd-3db14c73dfeb 2022-01-11T14:01:01.581 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.582+0000 7fa7a6a72500 20 RGWRados::pool_iterate: got zone_info.13be4714-7bb8-4369-8a42-f7f76c38c08c 2022-01-11T14:01:01.581 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.582+0000 7fa7a6a72500 20 RGWRados::pool_iterate: got zone_names.default 2022-01-11T14:01:01.582 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.582+0000 7fa7a6a72500 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-01-11T14:01:01.582 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.582+0000 7fa7a6a72500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.582 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.582+0000 7fa7a6a72500 20 rados_obj.operate() r=0 bl.length=873 2022-01-11T14:01:01.582 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.582+0000 7fa7a6a72500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.583 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.583+0000 7fa7a6a72500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:01.583 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.583+0000 7fa7a6a72500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.583 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.583+0000 7fa7a6a72500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:01.583 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.583+0000 7fa7a6a72500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.584 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.583+0000 7fa7a6a72500 20 rados_obj.operate() r=0 bl.length=362 2022-01-11T14:01:01.585 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:01:01.585 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "13be4714-7bb8-4369-8a42-f7f76c38c08c", 2022-01-11T14:01:01.585 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "default", 2022-01-11T14:01:01.586 INFO:teuthology.orchestra.run.smithi161.stdout: "domain_root": "default.rgw.meta:root", 2022-01-11T14:01:01.586 INFO:teuthology.orchestra.run.smithi161.stdout: "control_pool": "default.rgw.control", 2022-01-11T14:01:01.586 INFO:teuthology.orchestra.run.smithi161.stdout: "gc_pool": "default.rgw.log:gc", 2022-01-11T14:01:01.586 INFO:teuthology.orchestra.run.smithi161.stdout: "lc_pool": "default.rgw.log:lc", 2022-01-11T14:01:01.587 INFO:teuthology.orchestra.run.smithi161.stdout: "log_pool": "default.rgw.log", 2022-01-11T14:01:01.587 INFO:teuthology.orchestra.run.smithi161.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-01-11T14:01:01.587 INFO:teuthology.orchestra.run.smithi161.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-01-11T14:01:01.587 INFO:teuthology.orchestra.run.smithi161.stdout: "roles_pool": "default.rgw.meta:roles", 2022-01-11T14:01:01.588 INFO:teuthology.orchestra.run.smithi161.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-01-11T14:01:01.588 INFO:teuthology.orchestra.run.smithi161.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-01-11T14:01:01.588 INFO:teuthology.orchestra.run.smithi161.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-01-11T14:01:01.588 INFO:teuthology.orchestra.run.smithi161.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-01-11T14:01:01.588 INFO:teuthology.orchestra.run.smithi161.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-01-11T14:01:01.589 INFO:teuthology.orchestra.run.smithi161.stdout: "otp_pool": "default.rgw.otp", 2022-01-11T14:01:01.589 INFO:teuthology.orchestra.run.smithi161.stdout: "system_key": { 2022-01-11T14:01:01.589 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "", 2022-01-11T14:01:01.589 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "" 2022-01-11T14:01:01.590 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:01.590 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_pools": [ 2022-01-11T14:01:01.590 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:01.590 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "default-placement", 2022-01-11T14:01:01.590 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:01:01.591 INFO:teuthology.orchestra.run.smithi161.stdout: "index_pool": "default.rgw.buckets.index", 2022-01-11T14:01:01.591 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": { 2022-01-11T14:01:01.591 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD": { 2022-01-11T14:01:01.591 INFO:teuthology.orchestra.run.smithi161.stdout: "data_pool": "default.rgw.buckets.data", 2022-01-11T14:01:01.591 INFO:teuthology.orchestra.run.smithi161.stdout: "compression_type": "random" 2022-01-11T14:01:01.592 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:01.592 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:01.592 INFO:teuthology.orchestra.run.smithi161.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-01-11T14:01:01.592 INFO:teuthology.orchestra.run.smithi161.stdout: "index_type": 0 2022-01-11T14:01:01.593 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:01.593 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:01.593 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:01.593 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "", 2022-01-11T14:01:01.593 INFO:teuthology.orchestra.run.smithi161.stdout: "notif_pool": "default.rgw.log:notif" 2022-01-11T14:01:01.594 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:01:01.595 DEBUG:tasks.util.rgw: json result: {'id': '13be4714-7bb8-4369-8a42-f7f76c38c08c', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-01-11T14:01:01.595 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2022-01-11T14:01:01.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2022-01-11T14:01:01.596 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2022-01-11T14:01:01.634 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:01.634 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:01.639 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.641+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.642 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.642+0000 7f371bc30500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:01.642 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.642+0000 7f371bc30500 20 realm 2022-01-11T14:01:01.642 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.642+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.642 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.642+0000 7f371bc30500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.642+0000 7f371bc30500 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-01-11T14:01:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.642+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.642+0000 7f371bc30500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.642+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.643+0000 7f371bc30500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.643+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.643+0000 7f371bc30500 20 rados_obj.operate() r=0 bl.length=889 2022-01-11T14:01:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.644+0000 7f371bc30500 20 searching for the correct realm 2022-01-11T14:01:01.650 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.650+0000 7f371bc30500 20 RGWRados::pool_iterate: got zonegroup_info.6e25cdb6-f53f-43b0-83cd-3db14c73dfeb 2022-01-11T14:01:01.650 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.650+0000 7f371bc30500 20 RGWRados::pool_iterate: got zone_info.13be4714-7bb8-4369-8a42-f7f76c38c08c 2022-01-11T14:01:01.650 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.650+0000 7f371bc30500 20 RGWRados::pool_iterate: got zone_names.default 2022-01-11T14:01:01.652 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.650+0000 7f371bc30500 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-01-11T14:01:01.652 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.650+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.652 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.650+0000 7f371bc30500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:01.652 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.651+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.653 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.651+0000 7f371bc30500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:01.653 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.651+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.653 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.651+0000 7f371bc30500 20 rados_obj.operate() r=0 bl.length=362 2022-01-11T14:01:01.653 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.651+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.654 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.651+0000 7f371bc30500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:01.654 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.651+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.654 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.651+0000 7f371bc30500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:01.654 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.651+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.655 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.652+0000 7f371bc30500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:01.658 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.658+0000 7f371bc30500 20 RGWRados::pool_iterate: got zonegroup_info.6e25cdb6-f53f-43b0-83cd-3db14c73dfeb 2022-01-11T14:01:01.658 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.658+0000 7f371bc30500 20 RGWRados::pool_iterate: got zone_info.13be4714-7bb8-4369-8a42-f7f76c38c08c 2022-01-11T14:01:01.659 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.658+0000 7f371bc30500 20 RGWRados::pool_iterate: got zone_names.default 2022-01-11T14:01:01.659 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.658+0000 7f371bc30500 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-01-11T14:01:01.659 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.658+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.659 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.658+0000 7f371bc30500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:01.659 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.658+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.660 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.658+0000 7f371bc30500 20 rados_obj.operate() r=0 bl.length=889 2022-01-11T14:01:01.660 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.658+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.660 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.658+0000 7f371bc30500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:01.661 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.658+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.661 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.658+0000 7f371bc30500 20 rados_obj.operate() r=0 bl.length=362 2022-01-11T14:01:01.661 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.661 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:01.662 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 20 zone default found 2022-01-11T14:01:01.662 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 4 Realm: () 2022-01-11T14:01:01.662 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 4 ZoneGroup: default (6e25cdb6-f53f-43b0-83cd-3db14c73dfeb) 2022-01-11T14:01:01.662 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 4 Zone: default (13be4714-7bb8-4369-8a42-f7f76c38c08c) 2022-01-11T14:01:01.662 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 10 cannot find current period zonegroup using local zonegroup configuration 2022-01-11T14:01:01.663 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 20 zonegroup default 2022-01-11T14:01:01.663 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.663 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:01.663 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.664 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:01.664 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.664 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:01.664 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 20 started sync module instance, tier type = 2022-01-11T14:01:01.664 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.659+0000 7f371bc30500 20 started zone id=13be4714-7bb8-4369-8a42-f7f76c38c08c (name=default) with tier type = 2022-01-11T14:01:01.675 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.677+0000 7f371bc30500 20 add_watcher() i=0 2022-01-11T14:01:01.676 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.677+0000 7f371bc30500 20 add_watcher() i=1 2022-01-11T14:01:01.676 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.677+0000 7f371bc30500 20 add_watcher() i=2 2022-01-11T14:01:01.676 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.677+0000 7f371bc30500 20 add_watcher() i=3 2022-01-11T14:01:01.677 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.677+0000 7f371bc30500 20 add_watcher() i=4 2022-01-11T14:01:01.677 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.677+0000 7f371bc30500 20 add_watcher() i=5 2022-01-11T14:01:01.677 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.677+0000 7f371bc30500 20 add_watcher() i=6 2022-01-11T14:01:01.678 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.679+0000 7f371bc30500 20 add_watcher() i=7 2022-01-11T14:01:01.678 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.679+0000 7f371bc30500 2 all 8 watchers are set, enabling cache 2022-01-11T14:01:01.680 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.682+0000 7f36dffe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-01-11T14:01:01.681 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.682+0000 7f371bc30500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-01-11T14:01:01.681 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.682+0000 7f371bc30500 20 check_secure_mon_conn(): mode 1 is insecure 2022-01-11T14:01:01.684 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.686+0000 7f371bc30500 5 note: GC not initialized 2022-01-11T14:01:01.687 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.687+0000 7f364effd700 20 reqs_thread_entry: start 2022-01-11T14:01:01.687 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.687+0000 7f364cff9700 20 reqs_thread_entry: start 2022-01-11T14:01:01.688 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.687+0000 7f371bc30500 20 init_complete bucket index max shards: 11 2022-01-11T14:01:01.688 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.687+0000 7f371bc30500 10 rgw notify: Started notification manager with: 1 workers 2022-01-11T14:01:01.688 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.687+0000 7f3642ffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 11 14:01:32 2022 2022-01-11T14:01:01.689 INFO:teuthology.orchestra.run.smithi161.stderr: 2022-01-11T14:01:01.689 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.687+0000 7f36427fc700 20 reqs_thread_entry: start 2022-01-11T14:01:01.693 INFO:teuthology.orchestra.run.smithi161.stdout:[] 2022-01-11T14:01:01.694 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.695+0000 7f371bc30500 20 remove_watcher() i=0 2022-01-11T14:01:01.694 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.695+0000 7f371bc30500 2 removed watcher, disabling cache 2022-01-11T14:01:01.695 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.695+0000 7f371bc30500 20 remove_watcher() i=1 2022-01-11T14:01:01.696 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.696+0000 7f371bc30500 20 remove_watcher() i=2 2022-01-11T14:01:01.697 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.697+0000 7f371bc30500 20 remove_watcher() i=3 2022-01-11T14:01:01.697 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.698+0000 7f371bc30500 20 remove_watcher() i=4 2022-01-11T14:01:01.699 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.699+0000 7f371bc30500 20 remove_watcher() i=5 2022-01-11T14:01:01.699 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.700+0000 7f371bc30500 20 remove_watcher() i=6 2022-01-11T14:01:01.700 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.701+0000 7f371bc30500 20 remove_watcher() i=7 2022-01-11T14:01:01.701 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.701+0000 7f36dffe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-01-11T14:01:01.704 DEBUG:tasks.util.rgw: json result: [] 2022-01-11T14:01:01.705 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-01-11T14:01:01.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-01-11T14:01:01.706 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-01-11T14:01:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:01.788 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:01.793 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.795+0000 7fce70d68500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-01-11T14:01:01.793 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.795+0000 7fce70d68500 20 check_secure_mon_conn(): mode 1 is insecure 2022-01-11T14:01:01.797 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.798+0000 7fce70d68500 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-01-11T14:01:01.797 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.798+0000 7fcddd7fa700 20 reqs_thread_entry: start 2022-01-11T14:01:01.797 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.799+0000 7fce70d68500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.798 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.799+0000 7fce70d68500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:01.798 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.799+0000 7fce70d68500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.798 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.800+0000 7fce70d68500 20 rados_obj.operate() r=0 bl.length=889 2022-01-11T14:01:01.806 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.807+0000 7fce70d68500 20 RGWRados::pool_iterate: got zonegroup_info.6e25cdb6-f53f-43b0-83cd-3db14c73dfeb 2022-01-11T14:01:01.810 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.807+0000 7fce70d68500 20 RGWRados::pool_iterate: got zone_info.13be4714-7bb8-4369-8a42-f7f76c38c08c 2022-01-11T14:01:01.811 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.807+0000 7fce70d68500 20 RGWRados::pool_iterate: got zone_names.default 2022-01-11T14:01:01.811 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.807+0000 7fce70d68500 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-01-11T14:01:01.811 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.807+0000 7fce70d68500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.812 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.807+0000 7fce70d68500 20 rados_obj.operate() r=0 bl.length=889 2022-01-11T14:01:01.812 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.807+0000 7fce70d68500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.812 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.807+0000 7fce70d68500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:01.813 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.807+0000 7fce70d68500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.813 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.807+0000 7fce70d68500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:01.813 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.807+0000 7fce70d68500 20 rados->read ofs=0 len=0 2022-01-11T14:01:01.813 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:01.807+0000 7fce70d68500 20 rados_obj.operate() r=0 bl.length=362 2022-01-11T14:01:01.815 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:01:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "13be4714-7bb8-4369-8a42-f7f76c38c08c", 2022-01-11T14:01:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "default", 2022-01-11T14:01:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: "domain_root": "default.rgw.meta:root", 2022-01-11T14:01:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: "control_pool": "default.rgw.control", 2022-01-11T14:01:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: "gc_pool": "default.rgw.log:gc", 2022-01-11T14:01:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: "lc_pool": "default.rgw.log:lc", 2022-01-11T14:01:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: "log_pool": "default.rgw.log", 2022-01-11T14:01:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-01-11T14:01:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-01-11T14:01:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: "roles_pool": "default.rgw.meta:roles", 2022-01-11T14:01:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-01-11T14:01:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-01-11T14:01:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-01-11T14:01:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-01-11T14:01:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-01-11T14:01:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: "otp_pool": "default.rgw.otp", 2022-01-11T14:01:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: "system_key": { 2022-01-11T14:01:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "", 2022-01-11T14:01:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "" 2022-01-11T14:01:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_pools": [ 2022-01-11T14:01:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "default-placement", 2022-01-11T14:01:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:01:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: "index_pool": "default.rgw.buckets.index", 2022-01-11T14:01:01.823 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": { 2022-01-11T14:01:01.823 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD": { 2022-01-11T14:01:01.823 INFO:teuthology.orchestra.run.smithi161.stdout: "data_pool": "default.rgw.buckets.data", 2022-01-11T14:01:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: "compression_type": "random" 2022-01-11T14:01:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-01-11T14:01:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: "index_type": 0 2022-01-11T14:01:01.825 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:01.825 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:01.825 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "", 2022-01-11T14:01:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: "notif_pool": "default.rgw.log:notif" 2022-01-11T14:01:01.826 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:01:01.827 DEBUG:tasks.util.rgw: json result: {'id': '13be4714-7bb8-4369-8a42-f7f76c38c08c', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-01-11T14:01:01.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2022-01-11T14:01:01.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2022-01-11T14:01:01.828 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2022-01-11T14:01:01.940 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:01.941 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:01.946 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:01.948+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.904 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.904+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:03.904 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.904+0000 7f6757e29500 20 realm 2022-01-11T14:01:03.905 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.904+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.906 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.908+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:03.906 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.908+0000 7f6757e29500 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-01-11T14:01:03.906 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.908+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.907 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.909+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:03.908 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.909+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.909 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.910+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:03.909 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.910+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.909 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.910+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:03.910 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.910+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.910 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.910+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:03.910 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.910+0000 7f6757e29500 10 Creating default zonegroup 2022-01-11T14:01:03.910 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.910+0000 7f6757e29500 20 get_system_obj_state: rctx=0x7ffc28976730 obj=:.pools.avail state=0x56059e0e7c10 s->prefetch_data=0 2022-01-11T14:01:03.911 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.911+0000 7f6757e29500 10 couldn't find old data placement pools config, setting up new ones for the zone 2022-01-11T14:01:03.919 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.920+0000 7f6757e29500 10 WARNING: store->list_zones() returned r=-2 2022-01-11T14:01:03.919 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.920+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.919 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.920+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:03.921 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.923+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.922 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.923+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:03.922 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.923+0000 7f6757e29500 10 could not read realm id: (2) No such file or directory 2022-01-11T14:01:03.922 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.923+0000 7f6757e29500 10 WARNING: failed to set zone as default, r=-22 2022-01-11T14:01:03.923 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.923+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.923 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.923+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:03.925 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.926+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.926 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.927+0000 7f6757e29500 20 rados_obj.operate() r=0 bl.length=873 2022-01-11T14:01:03.926 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.927+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.926 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.927+0000 7f6757e29500 20 rados_obj.operate() r=0 bl.length=362 2022-01-11T14:01:03.926 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.927+0000 7f6757e29500 10 Using default name default 2022-01-11T14:01:03.927 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.927+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.928 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.928+0000 7f6757e29500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:03.928 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.928+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.928 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.928+0000 7f6757e29500 20 rados_obj.operate() r=0 bl.length=873 2022-01-11T14:01:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.928+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.928+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.928+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.928+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:03.929 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.928+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.930 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.929+0000 7f6757e29500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:03.935 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.936+0000 7f6757e29500 20 RGWRados::pool_iterate: got zonegroup_info.38fc78bd-63d7-4b7e-bcdc-f84a4f7c323e 2022-01-11T14:01:03.936 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.936+0000 7f6757e29500 20 RGWRados::pool_iterate: got zone_info.0764cabb-7625-47e2-a164-abdb3f2c9e6f 2022-01-11T14:01:03.936 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.936+0000 7f6757e29500 20 RGWRados::pool_iterate: got zone_names.default 2022-01-11T14:01:03.936 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.936+0000 7f6757e29500 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-01-11T14:01:03.936 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.936+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.936 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.937+0000 7f6757e29500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:03.937 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.937+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.937 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.937+0000 7f6757e29500 20 rados_obj.operate() r=0 bl.length=873 2022-01-11T14:01:03.937 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.937+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.937 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.937+0000 7f6757e29500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:03.937 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.937+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.939 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.937+0000 7f6757e29500 20 rados_obj.operate() r=0 bl.length=362 2022-01-11T14:01:03.939 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.937+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.939 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.938+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:03.939 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.938+0000 7f6757e29500 20 zone default found 2022-01-11T14:01:03.940 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.938+0000 7f6757e29500 4 Realm: () 2022-01-11T14:01:03.940 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.938+0000 7f6757e29500 4 ZoneGroup: default (38fc78bd-63d7-4b7e-bcdc-f84a4f7c323e) 2022-01-11T14:01:03.940 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.938+0000 7f6757e29500 4 Zone: default (0764cabb-7625-47e2-a164-abdb3f2c9e6f) 2022-01-11T14:01:03.940 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.938+0000 7f6757e29500 10 cannot find current period zonegroup using local zonegroup configuration 2022-01-11T14:01:03.941 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.938+0000 7f6757e29500 20 zonegroup default 2022-01-11T14:01:03.941 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.938+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:03.941 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.938+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:03.941 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:03.938+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:05.916 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:05.918+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:05.917 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:05.918+0000 7f6757e29500 20 rados->read ofs=0 len=0 2022-01-11T14:01:05.917 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:05.919+0000 7f6757e29500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:05.918 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:05.919+0000 7f6757e29500 20 started sync module instance, tier type = 2022-01-11T14:01:05.918 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:05.919+0000 7f6757e29500 20 started zone id=0764cabb-7625-47e2-a164-abdb3f2c9e6f (name=default) with tier type = 2022-01-11T14:01:07.951 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.952+0000 7f6757e29500 20 add_watcher() i=0 2022-01-11T14:01:07.952 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.952+0000 7f6757e29500 20 add_watcher() i=1 2022-01-11T14:01:07.952 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.952+0000 7f6757e29500 20 add_watcher() i=2 2022-01-11T14:01:07.952 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.952+0000 7f6757e29500 20 add_watcher() i=3 2022-01-11T14:01:07.952 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.952+0000 7f6757e29500 20 add_watcher() i=4 2022-01-11T14:01:07.952 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.952+0000 7f6757e29500 20 add_watcher() i=5 2022-01-11T14:01:07.953 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.952+0000 7f6757e29500 20 add_watcher() i=6 2022-01-11T14:01:07.955 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.954+0000 7f6757e29500 20 add_watcher() i=7 2022-01-11T14:01:07.955 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.954+0000 7f6757e29500 2 all 8 watchers are set, enabling cache 2022-01-11T14:01:07.955 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.954+0000 7f6757e29500 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2022-01-11T14:01:07.956 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.954+0000 7f6757e29500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2022-01-11T14:01:07.956 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.954+0000 7f6757e29500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-01-11T14:01:07.956 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.955+0000 7f6757e29500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2022-01-11T14:01:07.956 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.955+0000 7f6757e29500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2022-01-11T14:01:07.957 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.955+0000 7f6757e29500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-01-11T14:01:07.957 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.955+0000 7f6757e29500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2022-01-11T14:01:07.957 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.955+0000 7f6757e29500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2022-01-11T14:01:07.958 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.955+0000 7f6757e29500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-01-11T14:01:07.958 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.955+0000 7f6757e29500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2022-01-11T14:01:07.959 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.955+0000 7f6757e29500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2022-01-11T14:01:07.960 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.955+0000 7f6757e29500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-01-11T14:01:07.960 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.955+0000 7f6757e29500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2022-01-11T14:01:07.960 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.955+0000 7f6757e29500 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2022-01-11T14:01:07.961 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.956+0000 7f6757e29500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2022-01-11T14:01:07.962 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.959+0000 7f671bfe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-01-11T14:01:07.962 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.959+0000 7f6757e29500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-01-11T14:01:07.963 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:07.959+0000 7f6757e29500 20 check_secure_mon_conn(): mode 1 is insecure 2022-01-11T14:01:11.932 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.934+0000 7f6757e29500 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2022-01-11T14:01:11.933 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.934+0000 7f6757e29500 5 note: GC not initialized 2022-01-11T14:01:11.934 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.935+0000 7f6682ffd700 20 reqs_thread_entry: start 2022-01-11T14:01:11.935 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.936+0000 7f6680ff9700 20 reqs_thread_entry: start 2022-01-11T14:01:11.935 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.936+0000 7f6757e29500 20 init_complete bucket index max shards: 11 2022-01-11T14:01:11.935 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.936+0000 7f6757e29500 10 rgw notify: Started notification manager with: 1 workers 2022-01-11T14:01:11.936 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.936+0000 7f6672ffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 11 14:01:42 2022 2022-01-11T14:01:11.936 INFO:teuthology.orchestra.run.smithi172.stderr: 2022-01-11T14:01:11.936 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.936+0000 7f66727fc700 20 reqs_thread_entry: start 2022-01-11T14:01:11.959 INFO:teuthology.orchestra.run.smithi172.stdout:[] 2022-01-11T14:01:11.960 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.960+0000 7f6757e29500 20 remove_watcher() i=0 2022-01-11T14:01:11.960 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.960+0000 7f6757e29500 2 removed watcher, disabling cache 2022-01-11T14:01:11.960 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.961+0000 7f6757e29500 20 remove_watcher() i=1 2022-01-11T14:01:11.961 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.963+0000 7f6757e29500 20 remove_watcher() i=2 2022-01-11T14:01:11.963 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.965+0000 7f6757e29500 20 remove_watcher() i=3 2022-01-11T14:01:11.965 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.967+0000 7f6757e29500 20 remove_watcher() i=4 2022-01-11T14:01:11.967 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.969+0000 7f6757e29500 20 remove_watcher() i=5 2022-01-11T14:01:11.969 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.971+0000 7f6757e29500 20 remove_watcher() i=6 2022-01-11T14:01:11.970 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.972+0000 7f6757e29500 20 remove_watcher() i=7 2022-01-11T14:01:11.970 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:11.972+0000 7f671bfe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-01-11T14:01:11.975 DEBUG:tasks.util.rgw: json result: [] 2022-01-11T14:01:11.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-01-11T14:01:11.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-01-11T14:01:11.977 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-01-11T14:01:12.021 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:12.021 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:12.027 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.028+0000 7f5ef8645500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-01-11T14:01:12.027 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.028+0000 7f5ef8645500 20 check_secure_mon_conn(): mode 1 is insecure 2022-01-11T14:01:12.031 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.032+0000 7f5e8cff9700 20 reqs_thread_entry: start 2022-01-11T14:01:12.031 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.032+0000 7f5ef8645500 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-01-11T14:01:12.032 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.032+0000 7f5ef8645500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.032 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.032+0000 7f5ef8645500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:12.032 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.032+0000 7f5ef8645500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.032 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.033+0000 7f5ef8645500 20 rados_obj.operate() r=0 bl.length=873 2022-01-11T14:01:12.039 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.040+0000 7f5ef8645500 20 RGWRados::pool_iterate: got zonegroup_info.38fc78bd-63d7-4b7e-bcdc-f84a4f7c323e 2022-01-11T14:01:12.039 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.040+0000 7f5ef8645500 20 RGWRados::pool_iterate: got zone_info.0764cabb-7625-47e2-a164-abdb3f2c9e6f 2022-01-11T14:01:12.040 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.040+0000 7f5ef8645500 20 RGWRados::pool_iterate: got zone_names.default 2022-01-11T14:01:12.040 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.040+0000 7f5ef8645500 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-01-11T14:01:12.040 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.040+0000 7f5ef8645500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.040 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.040+0000 7f5ef8645500 20 rados_obj.operate() r=0 bl.length=873 2022-01-11T14:01:12.041 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.040+0000 7f5ef8645500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.041 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.040+0000 7f5ef8645500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:12.041 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.040+0000 7f5ef8645500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.041 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.041+0000 7f5ef8645500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:12.042 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.041+0000 7f5ef8645500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.042 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.041+0000 7f5ef8645500 20 rados_obj.operate() r=0 bl.length=362 2022-01-11T14:01:12.043 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:01:12.043 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "0764cabb-7625-47e2-a164-abdb3f2c9e6f", 2022-01-11T14:01:12.043 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default", 2022-01-11T14:01:12.044 INFO:teuthology.orchestra.run.smithi172.stdout: "domain_root": "default.rgw.meta:root", 2022-01-11T14:01:12.044 INFO:teuthology.orchestra.run.smithi172.stdout: "control_pool": "default.rgw.control", 2022-01-11T14:01:12.044 INFO:teuthology.orchestra.run.smithi172.stdout: "gc_pool": "default.rgw.log:gc", 2022-01-11T14:01:12.044 INFO:teuthology.orchestra.run.smithi172.stdout: "lc_pool": "default.rgw.log:lc", 2022-01-11T14:01:12.045 INFO:teuthology.orchestra.run.smithi172.stdout: "log_pool": "default.rgw.log", 2022-01-11T14:01:12.045 INFO:teuthology.orchestra.run.smithi172.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-01-11T14:01:12.045 INFO:teuthology.orchestra.run.smithi172.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-01-11T14:01:12.046 INFO:teuthology.orchestra.run.smithi172.stdout: "roles_pool": "default.rgw.meta:roles", 2022-01-11T14:01:12.046 INFO:teuthology.orchestra.run.smithi172.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-01-11T14:01:12.046 INFO:teuthology.orchestra.run.smithi172.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-01-11T14:01:12.046 INFO:teuthology.orchestra.run.smithi172.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-01-11T14:01:12.047 INFO:teuthology.orchestra.run.smithi172.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-01-11T14:01:12.047 INFO:teuthology.orchestra.run.smithi172.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-01-11T14:01:12.047 INFO:teuthology.orchestra.run.smithi172.stdout: "otp_pool": "default.rgw.otp", 2022-01-11T14:01:12.047 INFO:teuthology.orchestra.run.smithi172.stdout: "system_key": { 2022-01-11T14:01:12.048 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "", 2022-01-11T14:01:12.048 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "" 2022-01-11T14:01:12.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:01:12.048 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_pools": [ 2022-01-11T14:01:12.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:01:12.049 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "default-placement", 2022-01-11T14:01:12.049 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:01:12.049 INFO:teuthology.orchestra.run.smithi172.stdout: "index_pool": "default.rgw.buckets.index", 2022-01-11T14:01:12.049 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": { 2022-01-11T14:01:12.050 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD": { 2022-01-11T14:01:12.050 INFO:teuthology.orchestra.run.smithi172.stdout: "data_pool": "default.rgw.buckets.data", 2022-01-11T14:01:12.050 INFO:teuthology.orchestra.run.smithi172.stdout: "compression_type": "random" 2022-01-11T14:01:12.050 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:12.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:01:12.051 INFO:teuthology.orchestra.run.smithi172.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-01-11T14:01:12.051 INFO:teuthology.orchestra.run.smithi172.stdout: "index_type": 0 2022-01-11T14:01:12.051 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:12.051 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:12.052 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:01:12.052 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "", 2022-01-11T14:01:12.052 INFO:teuthology.orchestra.run.smithi172.stdout: "notif_pool": "default.rgw.log:notif" 2022-01-11T14:01:12.052 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:01:12.053 DEBUG:tasks.util.rgw: json result: {'id': '0764cabb-7625-47e2-a164-abdb3f2c9e6f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-01-11T14:01:12.054 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2022-01-11T14:01:12.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2022-01-11T14:01:12.055 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2022-01-11T14:01:12.132 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:12.132 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:12.137 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.138+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.140 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.139+0000 7fccec426500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:12.140 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.139+0000 7fccec426500 20 realm 2022-01-11T14:01:12.141 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.139+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.141 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.140+0000 7fccec426500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:12.141 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.140+0000 7fccec426500 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-01-11T14:01:12.142 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.140+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.142 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.140+0000 7fccec426500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:12.143 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.140+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.143 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.140+0000 7fccec426500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:12.144 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.140+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.144 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.141+0000 7fccec426500 20 rados_obj.operate() r=0 bl.length=889 2022-01-11T14:01:12.144 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.141+0000 7fccec426500 20 searching for the correct realm 2022-01-11T14:01:12.149 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.148+0000 7fccec426500 20 RGWRados::pool_iterate: got zonegroup_info.38fc78bd-63d7-4b7e-bcdc-f84a4f7c323e 2022-01-11T14:01:12.149 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.148+0000 7fccec426500 20 RGWRados::pool_iterate: got zone_info.0764cabb-7625-47e2-a164-abdb3f2c9e6f 2022-01-11T14:01:12.150 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.148+0000 7fccec426500 20 RGWRados::pool_iterate: got zone_names.default 2022-01-11T14:01:12.150 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.148+0000 7fccec426500 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-01-11T14:01:12.151 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.148+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.151 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.148+0000 7fccec426500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:12.151 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.148+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.152 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.148+0000 7fccec426500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:12.152 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.148+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.153 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.148+0000 7fccec426500 20 rados_obj.operate() r=0 bl.length=362 2022-01-11T14:01:12.153 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.148+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.153 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.148+0000 7fccec426500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:12.154 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.148+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.154 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.149+0000 7fccec426500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:12.155 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.149+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.156 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.149+0000 7fccec426500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:12.159 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.155+0000 7fccec426500 20 RGWRados::pool_iterate: got zonegroup_info.38fc78bd-63d7-4b7e-bcdc-f84a4f7c323e 2022-01-11T14:01:12.160 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.155+0000 7fccec426500 20 RGWRados::pool_iterate: got zone_info.0764cabb-7625-47e2-a164-abdb3f2c9e6f 2022-01-11T14:01:12.160 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.155+0000 7fccec426500 20 RGWRados::pool_iterate: got zone_names.default 2022-01-11T14:01:12.160 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.155+0000 7fccec426500 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-01-11T14:01:12.160 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.155+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.161 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.155+0000 7fccec426500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:12.161 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.155+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.161 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.155+0000 7fccec426500 20 rados_obj.operate() r=0 bl.length=889 2022-01-11T14:01:12.161 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.155+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.161 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.155+0000 7fccec426500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:12.162 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.155+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.162 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.156+0000 7fccec426500 20 rados_obj.operate() r=0 bl.length=362 2022-01-11T14:01:12.162 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.156+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.162 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.156+0000 7fccec426500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:12.162 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.156+0000 7fccec426500 20 zone default found 2022-01-11T14:01:12.163 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.156+0000 7fccec426500 4 Realm: () 2022-01-11T14:01:12.163 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.156+0000 7fccec426500 4 ZoneGroup: default (38fc78bd-63d7-4b7e-bcdc-f84a4f7c323e) 2022-01-11T14:01:12.163 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.156+0000 7fccec426500 4 Zone: default (0764cabb-7625-47e2-a164-abdb3f2c9e6f) 2022-01-11T14:01:12.163 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.156+0000 7fccec426500 10 cannot find current period zonegroup using local zonegroup configuration 2022-01-11T14:01:12.164 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.156+0000 7fccec426500 20 zonegroup default 2022-01-11T14:01:12.164 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.156+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.164 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.156+0000 7fccec426500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:12.164 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.156+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.164 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.156+0000 7fccec426500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:12.165 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.156+0000 7fccec426500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.165 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.157+0000 7fccec426500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:12.165 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.157+0000 7fccec426500 20 started sync module instance, tier type = 2022-01-11T14:01:12.165 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.157+0000 7fccec426500 20 started zone id=0764cabb-7625-47e2-a164-abdb3f2c9e6f (name=default) with tier type = 2022-01-11T14:01:12.166 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.167+0000 7fccec426500 20 add_watcher() i=0 2022-01-11T14:01:12.166 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.167+0000 7fccec426500 20 add_watcher() i=1 2022-01-11T14:01:12.167 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.167+0000 7fccec426500 20 add_watcher() i=2 2022-01-11T14:01:12.167 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.167+0000 7fccec426500 20 add_watcher() i=3 2022-01-11T14:01:12.167 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.167+0000 7fccec426500 20 add_watcher() i=4 2022-01-11T14:01:12.169 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.167+0000 7fccec426500 20 add_watcher() i=5 2022-01-11T14:01:12.169 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.168+0000 7fccec426500 20 add_watcher() i=6 2022-01-11T14:01:12.169 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.168+0000 7fccec426500 20 add_watcher() i=7 2022-01-11T14:01:12.170 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.168+0000 7fccec426500 2 all 8 watchers are set, enabling cache 2022-01-11T14:01:12.170 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.169+0000 7fccb0fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-01-11T14:01:12.171 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.170+0000 7fccec426500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-01-11T14:01:12.171 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.170+0000 7fccec426500 20 check_secure_mon_conn(): mode 1 is insecure 2022-01-11T14:01:12.171 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.173+0000 7fccec426500 5 note: GC not initialized 2022-01-11T14:01:12.173 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.174+0000 7fcc1f7fe700 20 reqs_thread_entry: start 2022-01-11T14:01:12.173 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.174+0000 7fcc1d7fa700 20 reqs_thread_entry: start 2022-01-11T14:01:12.173 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.174+0000 7fccec426500 20 init_complete bucket index max shards: 11 2022-01-11T14:01:12.174 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.174+0000 7fccec426500 10 rgw notify: Started notification manager with: 1 workers 2022-01-11T14:01:12.174 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.174+0000 7fcc0b7fe700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 11 14:01:42 2022 2022-01-11T14:01:12.174 INFO:teuthology.orchestra.run.smithi172.stderr: 2022-01-11T14:01:12.174 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.174+0000 7fcc0affd700 20 reqs_thread_entry: start 2022-01-11T14:01:12.180 INFO:teuthology.orchestra.run.smithi172.stdout:[] 2022-01-11T14:01:12.181 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.182+0000 7fccec426500 20 remove_watcher() i=0 2022-01-11T14:01:12.181 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.182+0000 7fccec426500 2 removed watcher, disabling cache 2022-01-11T14:01:12.181 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.183+0000 7fccec426500 20 remove_watcher() i=1 2022-01-11T14:01:12.182 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.184+0000 7fccec426500 20 remove_watcher() i=2 2022-01-11T14:01:12.183 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.185+0000 7fccec426500 20 remove_watcher() i=3 2022-01-11T14:01:12.183 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.185+0000 7fccec426500 20 remove_watcher() i=4 2022-01-11T14:01:12.185 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.186+0000 7fccec426500 20 remove_watcher() i=5 2022-01-11T14:01:12.185 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.187+0000 7fccec426500 20 remove_watcher() i=6 2022-01-11T14:01:12.186 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.188+0000 7fccec426500 20 remove_watcher() i=7 2022-01-11T14:01:12.187 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.189+0000 7fccb0fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-01-11T14:01:12.191 DEBUG:tasks.util.rgw: json result: [] 2022-01-11T14:01:12.192 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-01-11T14:01:12.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2022-01-11T14:01:12.192 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2022-01-11T14:01:12.274 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:12.274 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:12.279 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.281+0000 7fb6d1f19500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-01-11T14:01:12.279 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.281+0000 7fb6d1f19500 20 check_secure_mon_conn(): mode 1 is insecure 2022-01-11T14:01:12.283 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.284+0000 7fb60e7fc700 20 reqs_thread_entry: start 2022-01-11T14:01:12.283 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.284+0000 7fb6d1f19500 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2022-01-11T14:01:12.283 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.284+0000 7fb6d1f19500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.284 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.285+0000 7fb6d1f19500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:12.284 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.285+0000 7fb6d1f19500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.284 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.286+0000 7fb6d1f19500 20 rados_obj.operate() r=0 bl.length=889 2022-01-11T14:01:12.291 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.292+0000 7fb6d1f19500 20 RGWRados::pool_iterate: got zonegroup_info.38fc78bd-63d7-4b7e-bcdc-f84a4f7c323e 2022-01-11T14:01:12.291 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.292+0000 7fb6d1f19500 20 RGWRados::pool_iterate: got zone_info.0764cabb-7625-47e2-a164-abdb3f2c9e6f 2022-01-11T14:01:12.292 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.292+0000 7fb6d1f19500 20 RGWRados::pool_iterate: got zone_names.default 2022-01-11T14:01:12.292 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.292+0000 7fb6d1f19500 20 RGWRados::pool_iterate: got zonegroups_names.default 2022-01-11T14:01:12.292 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.292+0000 7fb6d1f19500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.292 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.292+0000 7fb6d1f19500 20 rados_obj.operate() r=0 bl.length=889 2022-01-11T14:01:12.293 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.293+0000 7fb6d1f19500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.293 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.293+0000 7fb6d1f19500 20 rados_obj.operate() r=-2 bl.length=0 2022-01-11T14:01:12.293 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.293+0000 7fb6d1f19500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.293 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.293+0000 7fb6d1f19500 20 rados_obj.operate() r=0 bl.length=46 2022-01-11T14:01:12.293 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.293+0000 7fb6d1f19500 20 rados->read ofs=0 len=0 2022-01-11T14:01:12.294 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:12.293+0000 7fb6d1f19500 20 rados_obj.operate() r=0 bl.length=362 2022-01-11T14:01:12.295 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:01:12.296 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "0764cabb-7625-47e2-a164-abdb3f2c9e6f", 2022-01-11T14:01:12.296 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default", 2022-01-11T14:01:12.296 INFO:teuthology.orchestra.run.smithi172.stdout: "domain_root": "default.rgw.meta:root", 2022-01-11T14:01:12.296 INFO:teuthology.orchestra.run.smithi172.stdout: "control_pool": "default.rgw.control", 2022-01-11T14:01:12.296 INFO:teuthology.orchestra.run.smithi172.stdout: "gc_pool": "default.rgw.log:gc", 2022-01-11T14:01:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: "lc_pool": "default.rgw.log:lc", 2022-01-11T14:01:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: "log_pool": "default.rgw.log", 2022-01-11T14:01:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: "intent_log_pool": "default.rgw.log:intent", 2022-01-11T14:01:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: "usage_log_pool": "default.rgw.log:usage", 2022-01-11T14:01:12.298 INFO:teuthology.orchestra.run.smithi172.stdout: "roles_pool": "default.rgw.meta:roles", 2022-01-11T14:01:12.298 INFO:teuthology.orchestra.run.smithi172.stdout: "reshard_pool": "default.rgw.log:reshard", 2022-01-11T14:01:12.298 INFO:teuthology.orchestra.run.smithi172.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2022-01-11T14:01:12.298 INFO:teuthology.orchestra.run.smithi172.stdout: "user_email_pool": "default.rgw.meta:users.email", 2022-01-11T14:01:12.298 INFO:teuthology.orchestra.run.smithi172.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2022-01-11T14:01:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2022-01-11T14:01:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: "otp_pool": "default.rgw.otp", 2022-01-11T14:01:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: "system_key": { 2022-01-11T14:01:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "", 2022-01-11T14:01:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "" 2022-01-11T14:01:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:01:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_pools": [ 2022-01-11T14:01:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:01:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "default-placement", 2022-01-11T14:01:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:01:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: "index_pool": "default.rgw.buckets.index", 2022-01-11T14:01:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": { 2022-01-11T14:01:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD": { 2022-01-11T14:01:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: "data_pool": "default.rgw.buckets.data", 2022-01-11T14:01:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: "compression_type": "random" 2022-01-11T14:01:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:01:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2022-01-11T14:01:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: "index_type": 0 2022-01-11T14:01:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:01:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "", 2022-01-11T14:01:12.304 INFO:teuthology.orchestra.run.smithi172.stdout: "notif_pool": "default.rgw.log:notif" 2022-01-11T14:01:12.304 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:01:12.305 DEBUG:tasks.util.rgw: json result: {'id': '0764cabb-7625-47e2-a164-abdb3f2c9e6f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2022-01-11T14:01:12.306 INFO:tasks.rgw:Starting rgw... 2022-01-11T14:01:12.306 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2022-01-11T14:01:12.306 INFO:tasks.rgw:Using beast as radosgw frontend 2022-01-11T14:01:12.307 INFO:tasks.rgw.c1.client.0:Restarting daemon 2022-01-11T14:01:12.307 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2022-01-11T14:01:12.310 INFO:tasks.rgw.c1.client.0:Started 2022-01-11T14:01:12.311 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2022-01-11T14:01:12.311 INFO:tasks.rgw:Using beast as radosgw frontend 2022-01-11T14:01:12.311 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-01-11T14:01:12.312 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-01-11T14:01:12.314 INFO:tasks.rgw.c1.client.1:Started 2022-01-11T14:01:12.315 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2022-01-11T14:01:12.315 INFO:tasks.rgw:Using beast as radosgw frontend 2022-01-11T14:01:12.315 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-01-11T14:01:12.316 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-01-11T14:01:12.347 INFO:tasks.rgw.c2.client.0:Started 2022-01-11T14:01:12.348 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2022-01-11T14:01:12.348 INFO:tasks.rgw:Using beast as radosgw frontend 2022-01-11T14:01:12.349 INFO:tasks.rgw.c2.client.1:Restarting daemon 2022-01-11T14:01:12.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2022-01-11T14:01:12.351 INFO:tasks.rgw.c2.client.1:Started 2022-01-11T14:01:12.352 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi161.front.sepia.ceph.com:8000/ 2022-01-11T14:01:12.352 DEBUG:teuthology.orchestra.run.smithi161:> curl http://smithi161.front.sepia.ceph.com:8000/ 2022-01-11T14:01:12.395 INFO:teuthology.orchestra.run.smithi161.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-01-11T14:01:12.396 INFO:teuthology.orchestra.run.smithi161.stderr: Dload Upload Total Spent Left Speed 2022-01-11T14:01:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-01-11T14:01:12.398 INFO:teuthology.orchestra.run.smithi161.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi161.front.sepia.ceph.com port 8000: Connection refused 2022-01-11T14:01:13.399 DEBUG:teuthology.orchestra.run.smithi161:> curl http://smithi161.front.sepia.ceph.com:8000/ 2022-01-11T14:01:13.419 INFO:teuthology.orchestra.run.smithi161.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-01-11T14:01:13.419 INFO:teuthology.orchestra.run.smithi161.stderr: Dload Upload Total Spent Left Speed 2022-01-11T14:01:13.419 INFO:teuthology.orchestra.run.smithi161.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-01-11T14:01:13.420 INFO:teuthology.orchestra.run.smithi161.stdout:anonymous 2022-01-11T14:01:13.421 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi161.front.sepia.ceph.com:8001/ 2022-01-11T14:01:13.421 DEBUG:teuthology.orchestra.run.smithi161:> curl http://smithi161.front.sepia.ceph.com:8001/ 2022-01-11T14:01:13.479 INFO:teuthology.orchestra.run.smithi161.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-01-11T14:01:13.480 INFO:teuthology.orchestra.run.smithi161.stderr: Dload Upload Total Spent Left Speed 2022-01-11T14:01:13.481 INFO:teuthology.orchestra.run.smithi161.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 104k 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 104k 2022-01-11T14:01:13.482 INFO:teuthology.orchestra.run.smithi161.stdout:anonymous 2022-01-11T14:01:13.483 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi172.front.sepia.ceph.com:8000/ 2022-01-11T14:01:13.483 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8000/ 2022-01-11T14:01:13.522 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-01-11T14:01:13.523 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2022-01-11T14:01:13.524 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-01-11T14:01:13.525 INFO:teuthology.orchestra.run.smithi172.stdout:anonymous 2022-01-11T14:01:13.525 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi172.front.sepia.ceph.com:8001/ 2022-01-11T14:01:13.527 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8001/ 2022-01-11T14:01:13.585 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-01-11T14:01:13.585 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2022-01-11T14:01:13.586 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-01-11T14:01:13.587 INFO:teuthology.orchestra.run.smithi172.stdout:anonymous 2022-01-11T14:01:13.587 INFO:teuthology.run_tasks:Running task rgw-multisite... 2022-01-11T14:01:13.806 INFO:tasks.rgw_multisite:creating realm.. 2022-01-11T14:01:13.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:13.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:13.808 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:01:13.847 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:13.847 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:13.880 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:01:13.880 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:01:13.880 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-realm", 2022-01-11T14:01:13.881 INFO:teuthology.orchestra.run.smithi161.stdout: "current_period": "bef33aea-c28e-4d3d-b156-c57e2ca93938", 2022-01-11T14:01:13.881 INFO:teuthology.orchestra.run.smithi161.stdout: "epoch": 1 2022-01-11T14:01:13.881 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:01:13.882 INFO:tasks.rgw_multisite:creating master zonegroup.. 2022-01-11T14:01:13.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:13.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:13.884 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:01:13.962 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:13.963 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:13.973 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:13.975+0000 7f58ae3c7500 -1 old format 2022-01-11T14:01:13.981 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:01:13.981 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "test-zonegroup", 2022-01-11T14:01:13.981 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zonegroup", 2022-01-11T14:01:13.981 INFO:teuthology.orchestra.run.smithi161.stdout: "api_name": "", 2022-01-11T14:01:13.982 INFO:teuthology.orchestra.run.smithi161.stdout: "is_master": "true", 2022-01-11T14:01:13.982 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:01:13.982 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:01:13.982 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:13.983 INFO:teuthology.orchestra.run.smithi161.stdout: "hostnames": [], 2022-01-11T14:01:13.983 INFO:teuthology.orchestra.run.smithi161.stdout: "hostnames_s3website": [], 2022-01-11T14:01:13.983 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zone": "", 2022-01-11T14:01:13.983 INFO:teuthology.orchestra.run.smithi161.stdout: "zones": [], 2022-01-11T14:01:13.984 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_targets": [], 2022-01-11T14:01:13.984 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "", 2022-01-11T14:01:13.984 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:01:13.984 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_policy": { 2022-01-11T14:01:13.984 INFO:teuthology.orchestra.run.smithi161.stdout: "groups": [] 2022-01-11T14:01:13.985 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:13.985 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:01:13.986 INFO:tasks.rgw_multisite:creating master zone.. 2022-01-11T14:01:13.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--endpoints', 'http://smithi161.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:13.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--endpoints', 'http://smithi161.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:13.987 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key QXCZ5A4RM0GLA2FE --secret Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi --endpoints http://smithi161.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:01:14.074 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:14.075 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:14.109 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:01:14.110 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:01:14.111 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone1", 2022-01-11T14:01:14.111 INFO:teuthology.orchestra.run.smithi161.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-01-11T14:01:14.111 INFO:teuthology.orchestra.run.smithi161.stdout: "control_pool": "test-zone1.rgw.control", 2022-01-11T14:01:14.111 INFO:teuthology.orchestra.run.smithi161.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-01-11T14:01:14.112 INFO:teuthology.orchestra.run.smithi161.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-01-11T14:01:14.112 INFO:teuthology.orchestra.run.smithi161.stdout: "log_pool": "test-zone1.rgw.log", 2022-01-11T14:01:14.112 INFO:teuthology.orchestra.run.smithi161.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-01-11T14:01:14.112 INFO:teuthology.orchestra.run.smithi161.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-01-11T14:01:14.112 INFO:teuthology.orchestra.run.smithi161.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-01-11T14:01:14.113 INFO:teuthology.orchestra.run.smithi161.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-01-11T14:01:14.113 INFO:teuthology.orchestra.run.smithi161.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-01-11T14:01:14.113 INFO:teuthology.orchestra.run.smithi161.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-01-11T14:01:14.113 INFO:teuthology.orchestra.run.smithi161.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-01-11T14:01:14.114 INFO:teuthology.orchestra.run.smithi161.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-01-11T14:01:14.114 INFO:teuthology.orchestra.run.smithi161.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-01-11T14:01:14.114 INFO:teuthology.orchestra.run.smithi161.stdout: "system_key": { 2022-01-11T14:01:14.114 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "QXCZ5A4RM0GLA2FE", 2022-01-11T14:01:14.115 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi" 2022-01-11T14:01:14.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:14.115 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_pools": [ 2022-01-11T14:01:14.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:14.115 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "default-placement", 2022-01-11T14:01:14.116 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:01:14.116 INFO:teuthology.orchestra.run.smithi161.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-01-11T14:01:14.116 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": { 2022-01-11T14:01:14.117 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD": { 2022-01-11T14:01:14.117 INFO:teuthology.orchestra.run.smithi161.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2022-01-11T14:01:14.117 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:14.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:14.118 INFO:teuthology.orchestra.run.smithi161.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-01-11T14:01:14.118 INFO:teuthology.orchestra.run.smithi161.stdout: "index_type": 0 2022-01-11T14:01:14.118 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:14.118 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:14.119 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:14.119 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:01:14.119 INFO:teuthology.orchestra.run.smithi161.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-01-11T14:01:14.119 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:01:14.121 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2022-01-11T14:01:15.211 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'test-zone1.rgw.buckets.data' created 2022-01-11T14:01:15.230 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2022-01-11T14:01:16.238 INFO:teuthology.orchestra.run.smithi161.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2022-01-11T14:01:16.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:16.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:16.253 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:01:16.300 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:16.301 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:16.324 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:01:16.324 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:01:16.325 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone1", 2022-01-11T14:01:16.325 INFO:teuthology.orchestra.run.smithi161.stdout: "domain_root": "test-zone1.rgw.meta:root", 2022-01-11T14:01:16.325 INFO:teuthology.orchestra.run.smithi161.stdout: "control_pool": "test-zone1.rgw.control", 2022-01-11T14:01:16.325 INFO:teuthology.orchestra.run.smithi161.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2022-01-11T14:01:16.326 INFO:teuthology.orchestra.run.smithi161.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2022-01-11T14:01:16.326 INFO:teuthology.orchestra.run.smithi161.stdout: "log_pool": "test-zone1.rgw.log", 2022-01-11T14:01:16.326 INFO:teuthology.orchestra.run.smithi161.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2022-01-11T14:01:16.326 INFO:teuthology.orchestra.run.smithi161.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2022-01-11T14:01:16.327 INFO:teuthology.orchestra.run.smithi161.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2022-01-11T14:01:16.327 INFO:teuthology.orchestra.run.smithi161.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2022-01-11T14:01:16.327 INFO:teuthology.orchestra.run.smithi161.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2022-01-11T14:01:16.327 INFO:teuthology.orchestra.run.smithi161.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2022-01-11T14:01:16.327 INFO:teuthology.orchestra.run.smithi161.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2022-01-11T14:01:16.328 INFO:teuthology.orchestra.run.smithi161.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2022-01-11T14:01:16.328 INFO:teuthology.orchestra.run.smithi161.stdout: "otp_pool": "test-zone1.rgw.otp", 2022-01-11T14:01:16.328 INFO:teuthology.orchestra.run.smithi161.stdout: "system_key": { 2022-01-11T14:01:16.328 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "QXCZ5A4RM0GLA2FE", 2022-01-11T14:01:16.329 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi" 2022-01-11T14:01:16.329 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:16.329 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_pools": [ 2022-01-11T14:01:16.329 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:16.330 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "default-placement", 2022-01-11T14:01:16.330 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:01:16.330 INFO:teuthology.orchestra.run.smithi161.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2022-01-11T14:01:16.330 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": { 2022-01-11T14:01:16.330 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD": { 2022-01-11T14:01:16.331 INFO:teuthology.orchestra.run.smithi161.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2022-01-11T14:01:16.331 INFO:teuthology.orchestra.run.smithi161.stdout: "compression_type": "random" 2022-01-11T14:01:16.331 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:16.331 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:16.332 INFO:teuthology.orchestra.run.smithi161.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2022-01-11T14:01:16.332 INFO:teuthology.orchestra.run.smithi161.stdout: "index_type": 0 2022-01-11T14:01:16.332 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:16.332 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:16.332 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:16.333 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "", 2022-01-11T14:01:16.333 INFO:teuthology.orchestra.run.smithi161.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2022-01-11T14:01:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:01:16.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:16.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:16.335 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:01:16.377 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:16.377 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:16.385 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:16.387+0000 7f7554e0f500 0 period (bef33aea-c28e-4d3d-b156-c57e2ca93938 does not have zone 776aa573-af61-46ca-a8eb-de18a933b620 configured 2022-01-11T14:01:24.293 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:01:24.294 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:01:24.294 INFO:teuthology.orchestra.run.smithi161.stdout: "epoch": 1, 2022-01-11T14:01:24.294 INFO:teuthology.orchestra.run.smithi161.stdout: "predecessor_uuid": "bef33aea-c28e-4d3d-b156-c57e2ca93938", 2022-01-11T14:01:24.295 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": [], 2022-01-11T14:01:24.295 INFO:teuthology.orchestra.run.smithi161.stdout: "period_map": { 2022-01-11T14:01:24.295 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:01:24.295 INFO:teuthology.orchestra.run.smithi161.stdout: "zonegroups": [ 2022-01-11T14:01:24.296 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:24.296 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "test-zonegroup", 2022-01-11T14:01:24.296 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zonegroup", 2022-01-11T14:01:24.296 INFO:teuthology.orchestra.run.smithi161.stdout: "api_name": "", 2022-01-11T14:01:24.297 INFO:teuthology.orchestra.run.smithi161.stdout: "is_master": "true", 2022-01-11T14:01:24.297 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:01:24.297 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:01:24.297 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:24.298 INFO:teuthology.orchestra.run.smithi161.stdout: "hostnames": [], 2022-01-11T14:01:24.298 INFO:teuthology.orchestra.run.smithi161.stdout: "hostnames_s3website": [], 2022-01-11T14:01:24.298 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zone": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:01:24.298 INFO:teuthology.orchestra.run.smithi161.stdout: "zones": [ 2022-01-11T14:01:24.299 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:24.299 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:01:24.299 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone1", 2022-01-11T14:01:24.299 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:01:24.299 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:01:24.300 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:24.300 INFO:teuthology.orchestra.run.smithi161.stdout: "log_meta": "false", 2022-01-11T14:01:24.300 INFO:teuthology.orchestra.run.smithi161.stdout: "log_data": "false", 2022-01-11T14:01:24.300 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:01:24.301 INFO:teuthology.orchestra.run.smithi161.stdout: "read_only": "false", 2022-01-11T14:01:24.301 INFO:teuthology.orchestra.run.smithi161.stdout: "tier_type": "", 2022-01-11T14:01:24.301 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from_all": "true", 2022-01-11T14:01:24.301 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from": [], 2022-01-11T14:01:24.302 INFO:teuthology.orchestra.run.smithi161.stdout: "redirect_zone": "" 2022-01-11T14:01:24.302 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:24.302 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:24.302 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_targets": [ 2022-01-11T14:01:24.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:24.303 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "default-placement", 2022-01-11T14:01:24.303 INFO:teuthology.orchestra.run.smithi161.stdout: "tags": [], 2022-01-11T14:01:24.303 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": [ 2022-01-11T14:01:24.304 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD" 2022-01-11T14:01:24.304 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:01:24.304 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:24.304 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:24.304 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "default-placement", 2022-01-11T14:01:24.305 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:01:24.305 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_policy": { 2022-01-11T14:01:24.305 INFO:teuthology.orchestra.run.smithi161.stdout: "groups": [] 2022-01-11T14:01:24.305 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:24.306 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:24.306 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:24.306 INFO:teuthology.orchestra.run.smithi161.stdout: "short_zone_ids": [ 2022-01-11T14:01:24.306 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:24.307 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:01:24.307 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 618671346 2022-01-11T14:01:24.307 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:24.307 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:01:24.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:24.308 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zonegroup": "test-zonegroup", 2022-01-11T14:01:24.308 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zone": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:01:24.308 INFO:teuthology.orchestra.run.smithi161.stdout: "period_config": { 2022-01-11T14:01:24.308 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_quota": { 2022-01-11T14:01:24.309 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2022-01-11T14:01:24.309 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2022-01-11T14:01:24.309 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2022-01-11T14:01:24.309 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2022-01-11T14:01:24.310 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2022-01-11T14:01:24.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:24.310 INFO:teuthology.orchestra.run.smithi161.stdout: "user_quota": { 2022-01-11T14:01:24.310 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2022-01-11T14:01:24.311 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2022-01-11T14:01:24.311 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2022-01-11T14:01:24.311 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2022-01-11T14:01:24.311 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2022-01-11T14:01:24.312 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:24.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:24.312 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:01:24.312 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_name": "test-realm", 2022-01-11T14:01:24.313 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:01:24.313 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:01:24.314 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2022-01-11T14:01:24.315 INFO:teuthology.orchestra.run:waiting for 300 2022-01-11T14:01:24.355 INFO:tasks.rgw.c1.client.0.smithi161.stdout:2022-01-11T14:01:24.358+0000 7f8e5b3fc700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 35312) UID: 0 2022-01-11T14:01:24.356 INFO:tasks.rgw.c1.client.0.smithi161.stdout:2022-01-11T14:01:24.358+0000 7f8e6c2845c0 -1 rgw dbstore: shutting down 2022-01-11T14:01:26.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2022-01-11T14:01:30.416 INFO:tasks.rgw.c1.client.0:Stopped 2022-01-11T14:01:30.417 INFO:tasks.rgw.c1.client.0:Restarting daemon 2022-01-11T14:01:30.417 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2022-01-11T14:01:30.420 INFO:tasks.rgw.c1.client.0:Started 2022-01-11T14:01:30.421 DEBUG:teuthology.orchestra.run.smithi161:> curl http://smithi161.front.sepia.ceph.com:8000 2022-01-11T14:01:30.441 INFO:teuthology.orchestra.run.smithi161.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-01-11T14:01:30.442 INFO:teuthology.orchestra.run.smithi161.stderr: Dload Upload Total Spent Left Speed 2022-01-11T14:01:30.442 INFO:teuthology.orchestra.run.smithi161.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi161.front.sepia.ceph.com port 8000: Connection refused 2022-01-11T14:01:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-01-11T14:01:31.445 DEBUG:teuthology.orchestra.run.smithi161:> curl http://smithi161.front.sepia.ceph.com:8000 2022-01-11T14:01:31.464 INFO:teuthology.orchestra.run.smithi161.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-01-11T14:01:31.465 INFO:teuthology.orchestra.run.smithi161.stderr: Dload Upload Total Spent Left Speed 2022-01-11T14:01:31.466 INFO:teuthology.orchestra.run.smithi161.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-01-11T14:01:31.467 INFO:teuthology.orchestra.run.smithi161.stdout:anonymous 2022-01-11T14:01:31.467 INFO:tasks.rgw_multisite:creating admin user.. 2022-01-11T14:01:31.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:31.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:31.468 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key QXCZ5A4RM0GLA2FE --secret Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:01:31.557 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:31.557 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:31.594 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:01:31.595 INFO:teuthology.orchestra.run.smithi161.stdout: "user_id": "realm-admin", 2022-01-11T14:01:31.595 INFO:teuthology.orchestra.run.smithi161.stdout: "display_name": "Realm Admin", 2022-01-11T14:01:31.595 INFO:teuthology.orchestra.run.smithi161.stdout: "email": "", 2022-01-11T14:01:31.595 INFO:teuthology.orchestra.run.smithi161.stdout: "suspended": 0, 2022-01-11T14:01:31.596 INFO:teuthology.orchestra.run.smithi161.stdout: "max_buckets": 1000, 2022-01-11T14:01:31.596 INFO:teuthology.orchestra.run.smithi161.stdout: "subusers": [], 2022-01-11T14:01:31.596 INFO:teuthology.orchestra.run.smithi161.stdout: "keys": [ 2022-01-11T14:01:31.596 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:31.597 INFO:teuthology.orchestra.run.smithi161.stdout: "user": "realm-admin", 2022-01-11T14:01:31.597 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "QXCZ5A4RM0GLA2FE", 2022-01-11T14:01:31.597 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi" 2022-01-11T14:01:31.597 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:31.597 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:31.598 INFO:teuthology.orchestra.run.smithi161.stdout: "swift_keys": [], 2022-01-11T14:01:31.598 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": [], 2022-01-11T14:01:31.598 INFO:teuthology.orchestra.run.smithi161.stdout: "op_mask": "read, write, delete", 2022-01-11T14:01:31.598 INFO:teuthology.orchestra.run.smithi161.stdout: "system": "true", 2022-01-11T14:01:31.599 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "", 2022-01-11T14:01:31.599 INFO:teuthology.orchestra.run.smithi161.stdout: "default_storage_class": "", 2022-01-11T14:01:31.599 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_tags": [], 2022-01-11T14:01:31.599 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_quota": { 2022-01-11T14:01:31.600 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2022-01-11T14:01:31.600 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2022-01-11T14:01:31.600 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2022-01-11T14:01:31.600 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2022-01-11T14:01:31.600 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2022-01-11T14:01:31.601 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:31.601 INFO:teuthology.orchestra.run.smithi161.stdout: "user_quota": { 2022-01-11T14:01:31.601 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2022-01-11T14:01:31.601 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2022-01-11T14:01:31.601 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2022-01-11T14:01:31.602 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2022-01-11T14:01:31.602 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2022-01-11T14:01:31.602 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:31.602 INFO:teuthology.orchestra.run.smithi161.stdout: "temp_url_keys": [], 2022-01-11T14:01:31.603 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "rgw", 2022-01-11T14:01:31.603 INFO:teuthology.orchestra.run.smithi161.stdout: "mfa_ids": [] 2022-01-11T14:01:31.603 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:01:31.603 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:01:31.611 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2022-01-11T14:01:31.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi161.front.sepia.ceph.com:8000', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:31.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi161.front.sepia.ceph.com:8000', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:31.612 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi161.front.sepia.ceph.com:8000 --access-key QXCZ5A4RM0GLA2FE --secret Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi --cluster c2 --debug-rgw 0 --debug-ms 0 2022-01-11T14:01:31.663 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:31.663 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:31.749 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:01:31.750 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:01:31.750 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-realm", 2022-01-11T14:01:31.750 INFO:teuthology.orchestra.run.smithi172.stdout: "current_period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:01:31.751 INFO:teuthology.orchestra.run.smithi172.stdout: "epoch": 2 2022-01-11T14:01:31.751 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:01:31.752 INFO:tasks.rgw_multisite:creating zone.. 2022-01-11T14:01:31.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--endpoints', 'http://smithi172.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:31.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--endpoints', 'http://smithi172.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:31.754 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key QXCZ5A4RM0GLA2FE --secret Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi --endpoints http://smithi172.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-01-11T14:01:31.803 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:31.803 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:31.828 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:31.830+0000 7f4287973500 0 failed reading obj info from .rgw.root:zone_info.776aa573-af61-46ca-a8eb-de18a933b620: (2) No such file or directory 2022-01-11T14:01:31.828 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:31.830+0000 7f4287973500 0 WARNING: could not read zone params for zone id=776aa573-af61-46ca-a8eb-de18a933b620 name=test-zone1 2022-01-11T14:01:31.832 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:01:31.832 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:01:31.833 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone2", 2022-01-11T14:01:31.833 INFO:teuthology.orchestra.run.smithi172.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-01-11T14:01:31.833 INFO:teuthology.orchestra.run.smithi172.stdout: "control_pool": "test-zone2.rgw.control", 2022-01-11T14:01:31.833 INFO:teuthology.orchestra.run.smithi172.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-01-11T14:01:31.834 INFO:teuthology.orchestra.run.smithi172.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-01-11T14:01:31.834 INFO:teuthology.orchestra.run.smithi172.stdout: "log_pool": "test-zone2.rgw.log", 2022-01-11T14:01:31.834 INFO:teuthology.orchestra.run.smithi172.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-01-11T14:01:31.834 INFO:teuthology.orchestra.run.smithi172.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-01-11T14:01:31.835 INFO:teuthology.orchestra.run.smithi172.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-01-11T14:01:31.835 INFO:teuthology.orchestra.run.smithi172.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-01-11T14:01:31.835 INFO:teuthology.orchestra.run.smithi172.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-01-11T14:01:31.835 INFO:teuthology.orchestra.run.smithi172.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-01-11T14:01:31.836 INFO:teuthology.orchestra.run.smithi172.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-01-11T14:01:31.836 INFO:teuthology.orchestra.run.smithi172.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-01-11T14:01:31.836 INFO:teuthology.orchestra.run.smithi172.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-01-11T14:01:31.836 INFO:teuthology.orchestra.run.smithi172.stdout: "system_key": { 2022-01-11T14:01:31.837 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "QXCZ5A4RM0GLA2FE", 2022-01-11T14:01:31.837 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi" 2022-01-11T14:01:31.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:01:31.837 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_pools": [ 2022-01-11T14:01:31.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:01:31.838 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "default-placement", 2022-01-11T14:01:31.838 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:01:31.838 INFO:teuthology.orchestra.run.smithi172.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-01-11T14:01:31.838 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": { 2022-01-11T14:01:31.839 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD": { 2022-01-11T14:01:31.839 INFO:teuthology.orchestra.run.smithi172.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2022-01-11T14:01:31.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:31.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:01:31.839 INFO:teuthology.orchestra.run.smithi172.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-01-11T14:01:31.840 INFO:teuthology.orchestra.run.smithi172.stdout: "index_type": 0 2022-01-11T14:01:31.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:31.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:31.840 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:01:31.841 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:01:31.841 INFO:teuthology.orchestra.run.smithi172.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-01-11T14:01:31.841 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:01:31.842 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2022-01-11T14:01:32.133 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2022-01-11T14:01:32.987 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'test-zone2.rgw.buckets.data' created 2022-01-11T14:01:33.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2022-01-11T14:01:34.005 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2022-01-11T14:01:34.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:34.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:34.019 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2022-01-11T14:01:34.068 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:34.068 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:34.092 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:01:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:01:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone2", 2022-01-11T14:01:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-01-11T14:01:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "control_pool": "test-zone2.rgw.control", 2022-01-11T14:01:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-01-11T14:01:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-01-11T14:01:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "log_pool": "test-zone2.rgw.log", 2022-01-11T14:01:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-01-11T14:01:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-01-11T14:01:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-01-11T14:01:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-01-11T14:01:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-01-11T14:01:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-01-11T14:01:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-01-11T14:01:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-01-11T14:01:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-01-11T14:01:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "system_key": { 2022-01-11T14:01:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "QXCZ5A4RM0GLA2FE", 2022-01-11T14:01:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi" 2022-01-11T14:01:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:01:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_pools": [ 2022-01-11T14:01:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:01:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "default-placement", 2022-01-11T14:01:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:01:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-01-11T14:01:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": { 2022-01-11T14:01:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD": { 2022-01-11T14:01:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-01-11T14:01:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "compression_type": "random" 2022-01-11T14:01:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:01:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-01-11T14:01:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "index_type": 0 2022-01-11T14:01:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:34.100 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:01:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "", 2022-01-11T14:01:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-01-11T14:01:34.101 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:01:34.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:34.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:34.104 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-01-11T14:01:34.145 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:34.145 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:34.154 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:01:34.155+0000 7f5fbaa1c500 0 period (86901227-b7e5-4fef-8f4e-54cce97cf2a2 does not have zone 3b4f5cd7-b080-4871-ae14-6286289db7bf configured 2022-01-11T14:01:42.064 INFO:teuthology.orchestra.run.smithi172.stderr:Sending period to new master zone 776aa573-af61-46ca-a8eb-de18a933b620 2022-01-11T14:01:42.111 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:01:42.112 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:01:42.113 INFO:teuthology.orchestra.run.smithi172.stdout: "epoch": 2, 2022-01-11T14:01:42.113 INFO:teuthology.orchestra.run.smithi172.stdout: "predecessor_uuid": "bef33aea-c28e-4d3d-b156-c57e2ca93938", 2022-01-11T14:01:42.113 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": [], 2022-01-11T14:01:42.114 INFO:teuthology.orchestra.run.smithi172.stdout: "period_map": { 2022-01-11T14:01:42.114 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:01:42.115 INFO:teuthology.orchestra.run.smithi172.stdout: "zonegroups": [ 2022-01-11T14:01:42.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:01:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "test-zonegroup", 2022-01-11T14:01:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zonegroup", 2022-01-11T14:01:42.117 INFO:teuthology.orchestra.run.smithi172.stdout: "api_name": "", 2022-01-11T14:01:42.117 INFO:teuthology.orchestra.run.smithi172.stdout: "is_master": "true", 2022-01-11T14:01:42.118 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2022-01-11T14:01:42.118 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:01:42.119 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:01:42.119 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames": [], 2022-01-11T14:01:42.119 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames_s3website": [], 2022-01-11T14:01:42.120 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:01:42.120 INFO:teuthology.orchestra.run.smithi172.stdout: "zones": [ 2022-01-11T14:01:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:01:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:01:42.122 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone2", 2022-01-11T14:01:42.122 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2022-01-11T14:01:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2022-01-11T14:01:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:01:42.124 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": "false", 2022-01-11T14:01:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": "true", 2022-01-11T14:01:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:01:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": "false", 2022-01-11T14:01:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2022-01-11T14:01:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": "true", 2022-01-11T14:01:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2022-01-11T14:01:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "" 2022-01-11T14:01:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:01:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:01:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:01:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone1", 2022-01-11T14:01:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2022-01-11T14:01:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:01:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:01:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": "false", 2022-01-11T14:01:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": "true", 2022-01-11T14:01:42.135 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:01:42.135 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": "false", 2022-01-11T14:01:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2022-01-11T14:01:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": "true", 2022-01-11T14:01:42.137 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2022-01-11T14:01:42.137 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "" 2022-01-11T14:01:42.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:42.138 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:01:42.139 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_targets": [ 2022-01-11T14:01:42.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:01:42.140 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default-placement", 2022-01-11T14:01:42.140 INFO:teuthology.orchestra.run.smithi172.stdout: "tags": [], 2022-01-11T14:01:42.140 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": [ 2022-01-11T14:01:42.141 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD" 2022-01-11T14:01:42.141 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:01:42.142 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:42.142 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:01:42.143 INFO:teuthology.orchestra.run.smithi172.stdout: "default_placement": "default-placement", 2022-01-11T14:01:42.143 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:01:42.144 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_policy": { 2022-01-11T14:01:42.144 INFO:teuthology.orchestra.run.smithi172.stdout: "groups": [] 2022-01-11T14:01:42.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:42.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:42.145 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:01:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: "short_zone_ids": [ 2022-01-11T14:01:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:01:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:01:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 2774529435 2022-01-11T14:01:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:01:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:01:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:01:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 618671346 2022-01-11T14:01:42.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:42.150 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:01:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:01:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zonegroup": "test-zonegroup", 2022-01-11T14:01:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:01:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "period_config": { 2022-01-11T14:01:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_quota": { 2022-01-11T14:01:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2022-01-11T14:01:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2022-01-11T14:01:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2022-01-11T14:01:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2022-01-11T14:01:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2022-01-11T14:01:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:01:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "user_quota": { 2022-01-11T14:01:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2022-01-11T14:01:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2022-01-11T14:01:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2022-01-11T14:01:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2022-01-11T14:01:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2022-01-11T14:01:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:01:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:01:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:01:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_name": "test-realm", 2022-01-11T14:01:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:01:42.161 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:01:42.164 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2022-01-11T14:01:42.164 INFO:teuthology.orchestra.run:waiting for 300 2022-01-11T14:01:42.205 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2022-01-11T14:01:42.206+0000 7fc3cd348700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 35345) UID: 0 2022-01-11T14:01:42.205 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2022-01-11T14:01:42.206+0000 7fc3de1d05c0 -1 rgw dbstore: shutting down 2022-01-11T14:01:42.953 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-01-11T14:01:48.267 INFO:tasks.rgw.c2.client.0:Stopped 2022-01-11T14:01:48.268 INFO:tasks.rgw.c2.client.0:Restarting daemon 2022-01-11T14:01:48.268 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2022-01-11T14:01:48.271 INFO:tasks.rgw.c2.client.0:Started 2022-01-11T14:01:48.272 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8000 2022-01-11T14:01:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-01-11T14:01:48.294 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-01-11T14:01:48.294 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2022-01-11T14:01:48.295 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi172.front.sepia.ceph.com port 8000: Connection refused 2022-01-11T14:01:48.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2022-01-11T14:01:49.296 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8000 2022-01-11T14:01:49.316 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-01-11T14:01:49.316 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2022-01-11T14:01:49.318 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-01-11T14:01:49.319 INFO:teuthology.orchestra.run.smithi172.stdout:anonymous 2022-01-11T14:01:49.319 INFO:tasks.rgw_multisite:creating zone.. 2022-01-11T14:01:49.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--endpoints', 'http://smithi161.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:49.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--endpoints', 'http://smithi161.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:49.321 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key QXCZ5A4RM0GLA2FE --secret Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi --endpoints http://smithi161.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:01:49.374 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:49.375 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:49.399 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:49.402+0000 7fc0e9530500 0 failed reading obj info from .rgw.root:zone_info.3b4f5cd7-b080-4871-ae14-6286289db7bf: (2) No such file or directory 2022-01-11T14:01:49.399 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:49.402+0000 7fc0e9530500 0 WARNING: could not read zone params for zone id=3b4f5cd7-b080-4871-ae14-6286289db7bf name=test-zone2 2022-01-11T14:01:49.404 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:01:49.404 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "a285481c-4c5c-487e-978c-9cbad192af6e", 2022-01-11T14:01:49.404 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone3", 2022-01-11T14:01:49.405 INFO:teuthology.orchestra.run.smithi161.stdout: "domain_root": "test-zone3.rgw.meta:root", 2022-01-11T14:01:49.405 INFO:teuthology.orchestra.run.smithi161.stdout: "control_pool": "test-zone3.rgw.control", 2022-01-11T14:01:49.405 INFO:teuthology.orchestra.run.smithi161.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2022-01-11T14:01:49.405 INFO:teuthology.orchestra.run.smithi161.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2022-01-11T14:01:49.406 INFO:teuthology.orchestra.run.smithi161.stdout: "log_pool": "test-zone3.rgw.log", 2022-01-11T14:01:49.406 INFO:teuthology.orchestra.run.smithi161.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2022-01-11T14:01:49.406 INFO:teuthology.orchestra.run.smithi161.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2022-01-11T14:01:49.406 INFO:teuthology.orchestra.run.smithi161.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2022-01-11T14:01:49.407 INFO:teuthology.orchestra.run.smithi161.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2022-01-11T14:01:49.407 INFO:teuthology.orchestra.run.smithi161.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2022-01-11T14:01:49.407 INFO:teuthology.orchestra.run.smithi161.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2022-01-11T14:01:49.407 INFO:teuthology.orchestra.run.smithi161.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2022-01-11T14:01:49.408 INFO:teuthology.orchestra.run.smithi161.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2022-01-11T14:01:49.408 INFO:teuthology.orchestra.run.smithi161.stdout: "otp_pool": "test-zone3.rgw.otp", 2022-01-11T14:01:49.408 INFO:teuthology.orchestra.run.smithi161.stdout: "system_key": { 2022-01-11T14:01:49.408 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "QXCZ5A4RM0GLA2FE", 2022-01-11T14:01:49.409 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi" 2022-01-11T14:01:49.409 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:49.409 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_pools": [ 2022-01-11T14:01:49.409 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:49.409 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "default-placement", 2022-01-11T14:01:49.410 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:01:49.410 INFO:teuthology.orchestra.run.smithi161.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2022-01-11T14:01:49.410 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": { 2022-01-11T14:01:49.410 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD": { 2022-01-11T14:01:49.411 INFO:teuthology.orchestra.run.smithi161.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2022-01-11T14:01:49.411 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:49.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:49.411 INFO:teuthology.orchestra.run.smithi161.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2022-01-11T14:01:49.412 INFO:teuthology.orchestra.run.smithi161.stdout: "index_type": 0 2022-01-11T14:01:49.412 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:49.412 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:49.412 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:49.412 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:01:49.413 INFO:teuthology.orchestra.run.smithi161.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2022-01-11T14:01:49.413 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:01:49.414 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2022-01-11T14:01:50.225 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'test-zone3.rgw.buckets.data' created 2022-01-11T14:01:50.240 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2022-01-11T14:01:51.228 INFO:teuthology.orchestra.run.smithi161.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2022-01-11T14:01:51.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:51.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:51.249 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:01:51.294 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:51.295 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:51.317 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:01:51.318 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "a285481c-4c5c-487e-978c-9cbad192af6e", 2022-01-11T14:01:51.318 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone3", 2022-01-11T14:01:51.318 INFO:teuthology.orchestra.run.smithi161.stdout: "domain_root": "test-zone3.rgw.meta:root", 2022-01-11T14:01:51.318 INFO:teuthology.orchestra.run.smithi161.stdout: "control_pool": "test-zone3.rgw.control", 2022-01-11T14:01:51.319 INFO:teuthology.orchestra.run.smithi161.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2022-01-11T14:01:51.319 INFO:teuthology.orchestra.run.smithi161.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2022-01-11T14:01:51.319 INFO:teuthology.orchestra.run.smithi161.stdout: "log_pool": "test-zone3.rgw.log", 2022-01-11T14:01:51.319 INFO:teuthology.orchestra.run.smithi161.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2022-01-11T14:01:51.320 INFO:teuthology.orchestra.run.smithi161.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2022-01-11T14:01:51.320 INFO:teuthology.orchestra.run.smithi161.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2022-01-11T14:01:51.320 INFO:teuthology.orchestra.run.smithi161.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2022-01-11T14:01:51.320 INFO:teuthology.orchestra.run.smithi161.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2022-01-11T14:01:51.321 INFO:teuthology.orchestra.run.smithi161.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2022-01-11T14:01:51.321 INFO:teuthology.orchestra.run.smithi161.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2022-01-11T14:01:51.321 INFO:teuthology.orchestra.run.smithi161.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2022-01-11T14:01:51.321 INFO:teuthology.orchestra.run.smithi161.stdout: "otp_pool": "test-zone3.rgw.otp", 2022-01-11T14:01:51.321 INFO:teuthology.orchestra.run.smithi161.stdout: "system_key": { 2022-01-11T14:01:51.322 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "QXCZ5A4RM0GLA2FE", 2022-01-11T14:01:51.322 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi" 2022-01-11T14:01:51.322 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:51.322 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_pools": [ 2022-01-11T14:01:51.323 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:51.323 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "default-placement", 2022-01-11T14:01:51.323 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:01:51.323 INFO:teuthology.orchestra.run.smithi161.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2022-01-11T14:01:51.324 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": { 2022-01-11T14:01:51.324 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD": { 2022-01-11T14:01:51.324 INFO:teuthology.orchestra.run.smithi161.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2022-01-11T14:01:51.324 INFO:teuthology.orchestra.run.smithi161.stdout: "compression_type": "random" 2022-01-11T14:01:51.325 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:51.325 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:51.325 INFO:teuthology.orchestra.run.smithi161.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2022-01-11T14:01:51.325 INFO:teuthology.orchestra.run.smithi161.stdout: "index_type": 0 2022-01-11T14:01:51.326 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:51.326 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:51.326 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:51.326 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "", 2022-01-11T14:01:51.326 INFO:teuthology.orchestra.run.smithi161.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2022-01-11T14:01:51.327 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:01:51.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:51.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:01:51.329 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:01:51.370 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:01:51.370 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:01:51.379 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:01:51.382+0000 7f7717707500 0 period (86901227-b7e5-4fef-8f4e-54cce97cf2a2 does not have zone a285481c-4c5c-487e-978c-9cbad192af6e configured 2022-01-11T14:01:59.296 INFO:teuthology.orchestra.run.smithi161.stderr:Sending period to new master zone 776aa573-af61-46ca-a8eb-de18a933b620 2022-01-11T14:01:59.322 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:01:59.322 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:01:59.323 INFO:teuthology.orchestra.run.smithi161.stdout: "epoch": 3, 2022-01-11T14:01:59.323 INFO:teuthology.orchestra.run.smithi161.stdout: "predecessor_uuid": "bef33aea-c28e-4d3d-b156-c57e2ca93938", 2022-01-11T14:01:59.323 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": [], 2022-01-11T14:01:59.323 INFO:teuthology.orchestra.run.smithi161.stdout: "period_map": { 2022-01-11T14:01:59.323 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:01:59.324 INFO:teuthology.orchestra.run.smithi161.stdout: "zonegroups": [ 2022-01-11T14:01:59.324 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:59.324 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "test-zonegroup", 2022-01-11T14:01:59.324 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zonegroup", 2022-01-11T14:01:59.325 INFO:teuthology.orchestra.run.smithi161.stdout: "api_name": "", 2022-01-11T14:01:59.325 INFO:teuthology.orchestra.run.smithi161.stdout: "is_master": "true", 2022-01-11T14:01:59.325 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:01:59.325 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:01:59.326 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:59.326 INFO:teuthology.orchestra.run.smithi161.stdout: "hostnames": [], 2022-01-11T14:01:59.326 INFO:teuthology.orchestra.run.smithi161.stdout: "hostnames_s3website": [], 2022-01-11T14:01:59.326 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zone": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:01:59.327 INFO:teuthology.orchestra.run.smithi161.stdout: "zones": [ 2022-01-11T14:01:59.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:59.327 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:01:59.327 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone2", 2022-01-11T14:01:59.327 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:01:59.328 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2022-01-11T14:01:59.328 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:59.328 INFO:teuthology.orchestra.run.smithi161.stdout: "log_meta": "false", 2022-01-11T14:01:59.328 INFO:teuthology.orchestra.run.smithi161.stdout: "log_data": "true", 2022-01-11T14:01:59.329 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:01:59.329 INFO:teuthology.orchestra.run.smithi161.stdout: "read_only": "false", 2022-01-11T14:01:59.329 INFO:teuthology.orchestra.run.smithi161.stdout: "tier_type": "", 2022-01-11T14:01:59.329 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from_all": "true", 2022-01-11T14:01:59.330 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from": [], 2022-01-11T14:01:59.330 INFO:teuthology.orchestra.run.smithi161.stdout: "redirect_zone": "" 2022-01-11T14:01:59.330 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:59.330 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:59.331 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:01:59.331 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone1", 2022-01-11T14:01:59.331 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:01:59.331 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:01:59.332 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:59.332 INFO:teuthology.orchestra.run.smithi161.stdout: "log_meta": "false", 2022-01-11T14:01:59.332 INFO:teuthology.orchestra.run.smithi161.stdout: "log_data": "true", 2022-01-11T14:01:59.332 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:01:59.332 INFO:teuthology.orchestra.run.smithi161.stdout: "read_only": "false", 2022-01-11T14:01:59.333 INFO:teuthology.orchestra.run.smithi161.stdout: "tier_type": "", 2022-01-11T14:01:59.333 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from_all": "true", 2022-01-11T14:01:59.333 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from": [], 2022-01-11T14:01:59.333 INFO:teuthology.orchestra.run.smithi161.stdout: "redirect_zone": "" 2022-01-11T14:01:59.334 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:59.334 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:59.334 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "a285481c-4c5c-487e-978c-9cbad192af6e", 2022-01-11T14:01:59.334 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone3", 2022-01-11T14:01:59.335 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:01:59.335 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi161.front.sepia.ceph.com:8001" 2022-01-11T14:01:59.335 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:59.335 INFO:teuthology.orchestra.run.smithi161.stdout: "log_meta": "false", 2022-01-11T14:01:59.336 INFO:teuthology.orchestra.run.smithi161.stdout: "log_data": "true", 2022-01-11T14:01:59.336 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:01:59.336 INFO:teuthology.orchestra.run.smithi161.stdout: "read_only": "false", 2022-01-11T14:01:59.336 INFO:teuthology.orchestra.run.smithi161.stdout: "tier_type": "", 2022-01-11T14:01:59.336 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from_all": "true", 2022-01-11T14:01:59.337 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from": [], 2022-01-11T14:01:59.337 INFO:teuthology.orchestra.run.smithi161.stdout: "redirect_zone": "" 2022-01-11T14:01:59.337 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:59.337 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:59.338 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_targets": [ 2022-01-11T14:01:59.338 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:59.338 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "default-placement", 2022-01-11T14:01:59.338 INFO:teuthology.orchestra.run.smithi161.stdout: "tags": [], 2022-01-11T14:01:59.338 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": [ 2022-01-11T14:01:59.339 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD" 2022-01-11T14:01:59.339 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:01:59.339 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:59.339 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:59.340 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "default-placement", 2022-01-11T14:01:59.340 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:01:59.340 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_policy": { 2022-01-11T14:01:59.340 INFO:teuthology.orchestra.run.smithi161.stdout: "groups": [] 2022-01-11T14:01:59.340 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:59.341 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:59.341 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:01:59.341 INFO:teuthology.orchestra.run.smithi161.stdout: "short_zone_ids": [ 2022-01-11T14:01:59.341 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:59.342 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:01:59.342 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 2774529435 2022-01-11T14:01:59.342 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:59.342 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:59.342 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:01:59.343 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 618671346 2022-01-11T14:01:59.343 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:59.343 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:01:59.343 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "a285481c-4c5c-487e-978c-9cbad192af6e", 2022-01-11T14:01:59.344 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 544948410 2022-01-11T14:01:59.344 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:59.344 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:01:59.344 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:59.345 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zonegroup": "test-zonegroup", 2022-01-11T14:01:59.345 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zone": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:01:59.345 INFO:teuthology.orchestra.run.smithi161.stdout: "period_config": { 2022-01-11T14:01:59.345 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_quota": { 2022-01-11T14:01:59.346 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2022-01-11T14:01:59.346 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2022-01-11T14:01:59.346 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2022-01-11T14:01:59.346 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2022-01-11T14:01:59.346 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2022-01-11T14:01:59.347 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:59.347 INFO:teuthology.orchestra.run.smithi161.stdout: "user_quota": { 2022-01-11T14:01:59.347 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2022-01-11T14:01:59.347 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2022-01-11T14:01:59.349 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2022-01-11T14:01:59.349 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2022-01-11T14:01:59.349 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2022-01-11T14:01:59.349 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:01:59.350 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:01:59.350 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:01:59.350 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_name": "test-realm", 2022-01-11T14:01:59.350 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:01:59.351 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:01:59.351 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2022-01-11T14:01:59.352 INFO:teuthology.orchestra.run:waiting for 300 2022-01-11T14:01:59.390 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:01:59.393+0000 7f54a5c70700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 35315) UID: 0 2022-01-11T14:01:59.390 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:01:59.393+0000 7f54b6af85c0 -1 rgw dbstore: shutting down 2022-01-11T14:02:00.579 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-01-11T14:02:05.454 INFO:tasks.rgw.c1.client.1:Stopped 2022-01-11T14:02:05.454 INFO:tasks.rgw.c1.client.1:Restarting daemon 2022-01-11T14:02:05.455 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2022-01-11T14:02:05.457 INFO:tasks.rgw.c1.client.1:Started 2022-01-11T14:02:05.458 DEBUG:teuthology.orchestra.run.smithi161:> curl http://smithi161.front.sepia.ceph.com:8001 2022-01-11T14:02:05.478 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-01-11T14:02:05.479 INFO:teuthology.orchestra.run.smithi161.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-01-11T14:02:05.479 INFO:teuthology.orchestra.run.smithi161.stderr: Dload Upload Total Spent Left Speed 2022-01-11T14:02:05.480 INFO:teuthology.orchestra.run.smithi161.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi161.front.sepia.ceph.com port 8001: Connection refused 2022-01-11T14:02:06.284 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2022-01-11T14:02:06.481 DEBUG:teuthology.orchestra.run.smithi161:> curl http://smithi161.front.sepia.ceph.com:8001 2022-01-11T14:02:06.501 INFO:teuthology.orchestra.run.smithi161.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-01-11T14:02:06.502 INFO:teuthology.orchestra.run.smithi161.stderr: Dload Upload Total Spent Left Speed 2022-01-11T14:02:06.502 INFO:teuthology.orchestra.run.smithi161.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2022-01-11T14:02:06.503 INFO:teuthology.orchestra.run.smithi161.stdout:anonymous 2022-01-11T14:02:06.504 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2022-01-11T14:02:06.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi161.front.sepia.ceph.com:8000', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--url', 'http://smithi161.front.sepia.ceph.com:8000', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:02:06.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi161.front.sepia.ceph.com:8000', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--url', 'http://smithi161.front.sepia.ceph.com:8000', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:02:06.505 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi161.front.sepia.ceph.com:8000 --access-key QXCZ5A4RM0GLA2FE --secret Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi --url http://smithi161.front.sepia.ceph.com:8000 --access-key QXCZ5A4RM0GLA2FE --secret Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi --cluster c2 --debug-rgw 0 --debug-ms 0 2022-01-11T14:02:06.563 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:02:06.563 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:03:53.559 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2022-01-11T14:03:53.560+0000 7f97a81e6700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669.control err=-107 2022-01-11T14:03:53.574 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:03:53.574 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:03:53.575 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-realm", 2022-01-11T14:03:53.575 INFO:teuthology.orchestra.run.smithi172.stdout: "current_period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:03:53.575 INFO:teuthology.orchestra.run.smithi172.stdout: "epoch": 2 2022-01-11T14:03:53.575 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:03:53.577 INFO:tasks.rgw_multisite:creating zone.. 2022-01-11T14:03:53.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--endpoints', 'http://smithi172.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:03:53.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'QXCZ5A4RM0GLA2FE', '--secret', 'Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi', '--endpoints', 'http://smithi172.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:03:53.579 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key QXCZ5A4RM0GLA2FE --secret Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi --endpoints http://smithi172.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2022-01-11T14:03:53.637 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:03:53.637 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:03:53.663 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:03:53.665+0000 7f2769de4500 0 failed reading obj info from .rgw.root:zone_info.776aa573-af61-46ca-a8eb-de18a933b620: (2) No such file or directory 2022-01-11T14:03:53.663 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:03:53.665+0000 7f2769de4500 0 WARNING: could not read zone params for zone id=776aa573-af61-46ca-a8eb-de18a933b620 name=test-zone1 2022-01-11T14:03:53.664 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:03:53.665+0000 7f2769de4500 0 failed reading obj info from .rgw.root:zone_info.a285481c-4c5c-487e-978c-9cbad192af6e: (2) No such file or directory 2022-01-11T14:03:53.664 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:03:53.665+0000 7f2769de4500 0 WARNING: could not read zone params for zone id=a285481c-4c5c-487e-978c-9cbad192af6e name=test-zone3 2022-01-11T14:03:53.667 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:03:53.667 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "ccb5abc6-5490-49c7-b386-02859e29748c", 2022-01-11T14:03:53.668 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone4", 2022-01-11T14:03:53.668 INFO:teuthology.orchestra.run.smithi172.stdout: "domain_root": "test-zone4.rgw.meta:root", 2022-01-11T14:03:53.668 INFO:teuthology.orchestra.run.smithi172.stdout: "control_pool": "test-zone4.rgw.control", 2022-01-11T14:03:53.668 INFO:teuthology.orchestra.run.smithi172.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2022-01-11T14:03:53.669 INFO:teuthology.orchestra.run.smithi172.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2022-01-11T14:03:53.669 INFO:teuthology.orchestra.run.smithi172.stdout: "log_pool": "test-zone4.rgw.log", 2022-01-11T14:03:53.669 INFO:teuthology.orchestra.run.smithi172.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2022-01-11T14:03:53.669 INFO:teuthology.orchestra.run.smithi172.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2022-01-11T14:03:53.669 INFO:teuthology.orchestra.run.smithi172.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2022-01-11T14:03:53.670 INFO:teuthology.orchestra.run.smithi172.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2022-01-11T14:03:53.670 INFO:teuthology.orchestra.run.smithi172.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2022-01-11T14:03:53.670 INFO:teuthology.orchestra.run.smithi172.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2022-01-11T14:03:53.671 INFO:teuthology.orchestra.run.smithi172.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2022-01-11T14:03:53.671 INFO:teuthology.orchestra.run.smithi172.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2022-01-11T14:03:53.671 INFO:teuthology.orchestra.run.smithi172.stdout: "otp_pool": "test-zone4.rgw.otp", 2022-01-11T14:03:53.671 INFO:teuthology.orchestra.run.smithi172.stdout: "system_key": { 2022-01-11T14:03:53.671 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "QXCZ5A4RM0GLA2FE", 2022-01-11T14:03:53.672 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi" 2022-01-11T14:03:53.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:03:53.672 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_pools": [ 2022-01-11T14:03:53.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:03:53.673 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "default-placement", 2022-01-11T14:03:53.673 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:03:53.673 INFO:teuthology.orchestra.run.smithi172.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2022-01-11T14:03:53.673 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": { 2022-01-11T14:03:53.674 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD": { 2022-01-11T14:03:53.674 INFO:teuthology.orchestra.run.smithi172.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2022-01-11T14:03:53.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:03:53.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:03:53.675 INFO:teuthology.orchestra.run.smithi172.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2022-01-11T14:03:53.675 INFO:teuthology.orchestra.run.smithi172.stdout: "index_type": 0 2022-01-11T14:03:53.675 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:03:53.675 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:03:53.676 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:03:53.676 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_config": { 2022-01-11T14:03:53.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event_retention_days": 7, 2022-01-11T14:03:53.676 INFO:teuthology.orchestra.run.smithi172.stdout: "start_with_full_sync": false 2022-01-11T14:03:53.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:03:53.677 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:03:53.677 INFO:teuthology.orchestra.run.smithi172.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2022-01-11T14:03:53.677 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:03:53.679 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2022-01-11T14:03:54.959 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'test-zone4.rgw.buckets.data' created 2022-01-11T14:03:54.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2022-01-11T14:03:55.971 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2022-01-11T14:03:55.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:03:55.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:03:55.998 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2022-01-11T14:03:56.045 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:03:56.046 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:03:56.069 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:03:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "ccb5abc6-5490-49c7-b386-02859e29748c", 2022-01-11T14:03:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone4", 2022-01-11T14:03:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "domain_root": "test-zone4.rgw.meta:root", 2022-01-11T14:03:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "control_pool": "test-zone4.rgw.control", 2022-01-11T14:03:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2022-01-11T14:03:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2022-01-11T14:03:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "log_pool": "test-zone4.rgw.log", 2022-01-11T14:03:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2022-01-11T14:03:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2022-01-11T14:03:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2022-01-11T14:03:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2022-01-11T14:03:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2022-01-11T14:03:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2022-01-11T14:03:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2022-01-11T14:03:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2022-01-11T14:03:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "otp_pool": "test-zone4.rgw.otp", 2022-01-11T14:03:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "system_key": { 2022-01-11T14:03:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "QXCZ5A4RM0GLA2FE", 2022-01-11T14:03:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi" 2022-01-11T14:03:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:03:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_pools": [ 2022-01-11T14:03:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:03:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "default-placement", 2022-01-11T14:03:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:03:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2022-01-11T14:03:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": { 2022-01-11T14:03:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD": { 2022-01-11T14:03:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2022-01-11T14:03:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "compression_type": "random" 2022-01-11T14:03:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:03:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:03:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2022-01-11T14:03:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "index_type": 0 2022-01-11T14:03:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:03:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:03:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:03:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_config": { 2022-01-11T14:03:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event_retention_days": 7, 2022-01-11T14:03:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "start_with_full_sync": false 2022-01-11T14:03:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:03:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "", 2022-01-11T14:03:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2022-01-11T14:03:56.080 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:03:56.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:03:56.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:03:56.082 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-01-11T14:03:56.123 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:03:56.124 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:03:56.132 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:03:56.134+0000 7fbf65587500 0 period (86901227-b7e5-4fef-8f4e-54cce97cf2a2 does not have zone ccb5abc6-5490-49c7-b386-02859e29748c configured 2022-01-11T14:04:04.046 INFO:teuthology.orchestra.run.smithi172.stderr:Sending period to new master zone 776aa573-af61-46ca-a8eb-de18a933b620 2022-01-11T14:04:04.072 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:04:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:04:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: "epoch": 4, 2022-01-11T14:04:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: "predecessor_uuid": "bef33aea-c28e-4d3d-b156-c57e2ca93938", 2022-01-11T14:04:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": [], 2022-01-11T14:04:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: "period_map": { 2022-01-11T14:04:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:04:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: "zonegroups": [ 2022-01-11T14:04:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "test-zonegroup", 2022-01-11T14:04:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zonegroup", 2022-01-11T14:04:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: "api_name": "", 2022-01-11T14:04:04.075 INFO:teuthology.orchestra.run.smithi172.stdout: "is_master": "true", 2022-01-11T14:04:04.075 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2022-01-11T14:04:04.075 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:04:04.075 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:04:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames": [], 2022-01-11T14:04:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames_s3website": [], 2022-01-11T14:04:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:04:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: "zones": [ 2022-01-11T14:04:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:04.077 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:04:04.077 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone2", 2022-01-11T14:04:04.077 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2022-01-11T14:04:04.077 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2022-01-11T14:04:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:04:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": "false", 2022-01-11T14:04:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": "true", 2022-01-11T14:04:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:04:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": "false", 2022-01-11T14:04:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2022-01-11T14:04:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": "true", 2022-01-11T14:04:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2022-01-11T14:04:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "" 2022-01-11T14:04:04.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:04.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:04.080 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:04:04.080 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone1", 2022-01-11T14:04:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2022-01-11T14:04:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:04:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:04:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": "false", 2022-01-11T14:04:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": "true", 2022-01-11T14:04:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:04:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": "false", 2022-01-11T14:04:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2022-01-11T14:04:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": "true", 2022-01-11T14:04:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2022-01-11T14:04:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "" 2022-01-11T14:04:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "a285481c-4c5c-487e-978c-9cbad192af6e", 2022-01-11T14:04:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone3", 2022-01-11T14:04:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2022-01-11T14:04:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi161.front.sepia.ceph.com:8001" 2022-01-11T14:04:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:04:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": "false", 2022-01-11T14:04:04.086 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": "true", 2022-01-11T14:04:04.086 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:04:04.086 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": "false", 2022-01-11T14:04:04.086 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2022-01-11T14:04:04.087 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": "true", 2022-01-11T14:04:04.087 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2022-01-11T14:04:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "" 2022-01-11T14:04:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "ccb5abc6-5490-49c7-b386-02859e29748c", 2022-01-11T14:04:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone4", 2022-01-11T14:04:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2022-01-11T14:04:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2022-01-11T14:04:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:04:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": "false", 2022-01-11T14:04:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": "true", 2022-01-11T14:04:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:04:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": "false", 2022-01-11T14:04:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "pubsub", 2022-01-11T14:04:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": "false", 2022-01-11T14:04:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [ 2022-01-11T14:04:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: "test-zone1" 2022-01-11T14:04:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:04:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "" 2022-01-11T14:04:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:04:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_targets": [ 2022-01-11T14:04:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default-placement", 2022-01-11T14:04:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: "tags": [], 2022-01-11T14:04:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": [ 2022-01-11T14:04:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD" 2022-01-11T14:04:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:04:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:04:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: "default_placement": "default-placement", 2022-01-11T14:04:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:04:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_policy": { 2022-01-11T14:04:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: "groups": [] 2022-01-11T14:04:04.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:04.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:04.096 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:04:04.096 INFO:teuthology.orchestra.run.smithi172.stdout: "short_zone_ids": [ 2022-01-11T14:04:04.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:04.098 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:04:04.098 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 2774529435 2022-01-11T14:04:04.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:04.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:04.099 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:04:04.099 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 618671346 2022-01-11T14:04:04.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:04.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:04.100 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "a285481c-4c5c-487e-978c-9cbad192af6e", 2022-01-11T14:04:04.100 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 544948410 2022-01-11T14:04:04.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:04.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:04.101 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "ccb5abc6-5490-49c7-b386-02859e29748c", 2022-01-11T14:04:04.101 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 2309622666 2022-01-11T14:04:04.101 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:04.101 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:04:04.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:04.102 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zonegroup": "test-zonegroup", 2022-01-11T14:04:04.102 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:04:04.102 INFO:teuthology.orchestra.run.smithi172.stdout: "period_config": { 2022-01-11T14:04:04.103 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_quota": { 2022-01-11T14:04:04.103 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2022-01-11T14:04:04.103 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2022-01-11T14:04:04.103 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2022-01-11T14:04:04.103 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2022-01-11T14:04:04.104 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2022-01-11T14:04:04.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:04.104 INFO:teuthology.orchestra.run.smithi172.stdout: "user_quota": { 2022-01-11T14:04:04.104 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2022-01-11T14:04:04.105 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2022-01-11T14:04:04.105 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2022-01-11T14:04:04.105 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2022-01-11T14:04:04.105 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2022-01-11T14:04:04.105 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:04.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:04.106 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:04:04.106 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_name": "test-realm", 2022-01-11T14:04:04.106 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:04.107 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:04:04.107 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2022-01-11T14:04:04.108 INFO:teuthology.orchestra.run:waiting for 300 2022-01-11T14:04:04.139 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:04:04.142+0000 7feede44a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 35349) UID: 0 2022-01-11T14:04:04.140 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:04:04.142+0000 7feeef2d25c0 -1 rgw dbstore: shutting down 2022-01-11T14:04:05.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2022-01-11T14:04:10.209 INFO:tasks.rgw.c2.client.1:Stopped 2022-01-11T14:04:10.210 INFO:tasks.rgw.c2.client.1:Restarting daemon 2022-01-11T14:04:10.211 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2022-01-11T14:04:10.214 INFO:tasks.rgw.c2.client.1:Started 2022-01-11T14:04:10.214 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8001 2022-01-11T14:04:10.235 DEBUG:teuthology.orchestra.run:got remote process result: 7 2022-01-11T14:04:10.236 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-01-11T14:04:10.236 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2022-01-11T14:04:10.237 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi172.front.sepia.ceph.com port 8001: Connection refused 2022-01-11T14:04:11.238 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8001 2022-01-11T14:04:11.260 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-01-11T14:04:11.260 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2022-01-11T14:04:11.260 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 130k 0 --:--:-- --:--:-- --:--:-- 130k 2022-01-11T14:04:11.261 INFO:teuthology.orchestra.run.smithi172.stdout:{"Code":"MethodNotAllowed","RequestId":"tx00000d2fdbc9254dae2c2-0061dd8e5b-10ea-test-zone4","HostId":"10ea-test-zone4-test-zonegroup"} 2022-01-11T14:04:11.262 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2022-01-11T14:04:11.262 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2022-01-11T14:04:11.279 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_17f142526478a11e8f62d3933be552d546c4fd19/qa/../src/test/rgw/rgw_multi 2022-01-11T14:04:11.296 INFO:tasks.rgw_multisite_tests:creating test user.. 2022-01-11T14:04:11.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:04:11.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:04:11.297 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:04:11.323 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2022-01-11T14:04:11.351 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:04:11.352 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:04:11.418 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:04:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: "user_id": "rgw-multisite-test-user", 2022-01-11T14:04:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: "display_name": "Multisite Test User", 2022-01-11T14:04:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: "email": "", 2022-01-11T14:04:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: "suspended": 0, 2022-01-11T14:04:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: "max_buckets": 1000, 2022-01-11T14:04:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: "subusers": [], 2022-01-11T14:04:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: "keys": [ 2022-01-11T14:04:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:04:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: "user": "rgw-multisite-test-user", 2022-01-11T14:04:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "GD513DUFZFDGTLAK1JYN", 2022-01-11T14:04:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "FDRhDJ86ANTZSfrGo0ooUvgef2aftU2h4Sqsts9R" 2022-01-11T14:04:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:04:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:04:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: "swift_keys": [], 2022-01-11T14:04:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": [], 2022-01-11T14:04:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: "op_mask": "read, write, delete", 2022-01-11T14:04:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "", 2022-01-11T14:04:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: "default_storage_class": "", 2022-01-11T14:04:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_tags": [], 2022-01-11T14:04:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_quota": { 2022-01-11T14:04:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2022-01-11T14:04:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2022-01-11T14:04:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2022-01-11T14:04:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2022-01-11T14:04:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2022-01-11T14:04:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:04:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: "user_quota": { 2022-01-11T14:04:11.425 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2022-01-11T14:04:11.425 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2022-01-11T14:04:11.425 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2022-01-11T14:04:11.425 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2022-01-11T14:04:11.426 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2022-01-11T14:04:11.426 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:04:11.426 INFO:teuthology.orchestra.run.smithi161.stdout: "temp_url_keys": [], 2022-01-11T14:04:11.426 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "rgw", 2022-01-11T14:04:11.426 INFO:teuthology.orchestra.run.smithi161.stdout: "mfa_ids": [] 2022-01-11T14:04:11.427 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:04:11.427 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-01-11T14:04:11.438 INFO:rgw_multi.tests:meta checkpoint 2022-01-11T14:04:11.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:11.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:11.440 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:04:11.481 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:04:11.481 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:04:11.530 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:04:11.545 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:04:11.546 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:04:11.546 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:04:11.546 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:04:11.416936Z" 2022-01-11T14:04:11.547 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:04:11.547 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:04:11.547 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:04:11.548 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:04:11.548 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:04:11.548 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:04:11.548 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:04:11.548 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:04:11.549 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:04:11.549 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:04:11.549 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:04:11.549 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:04:11.550 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:04:11.550 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:04:11.551 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:04:11.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:11.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:11.553 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:04:11.595 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:04:11.595 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:04:11.658 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:04:11.658 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:04:11.659 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:04:11.659 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:04:11.660 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:04:11.660 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:04:11.660 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:11.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:11.661 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:04:11.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:11.661 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:04:11.661 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:11.661 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:11.662 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:11.662 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:11.662 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:04:11.662 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:11.663 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:11.663 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:11.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:11.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:11.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:11.664 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:04:11.664 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:11.665 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:11.665 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:11.665 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:11.665 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:11.665 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:11.666 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:11.666 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:11.666 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:11.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:11.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:11.667 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:04:11.667 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:11.667 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:11.668 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:11.668 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:11.668 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:11.668 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:11.668 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:11.669 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:11.669 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:11.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:11.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:11.670 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:04:11.670 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:11.670 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:11.670 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:11.671 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:11.671 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:11.671 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:11.672 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:11.672 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:11.672 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:11.672 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:11.673 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:04:11.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:11.673 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:04:11.673 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:04:11.673 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:04:11.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:11.674 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:04:11.675 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:04:16.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:16.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:16.678 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:04:16.727 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:04:16.728 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:04:16.780 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:04:16.780 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:04:16.781 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:04:16.781 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:04:16.781 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:04:16.781 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:04:16.781 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:16.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:16.782 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:04:16.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:16.782 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:04:16.783 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:16.783 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:16.783 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:16.783 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:16.783 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:04:16.784 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:16.784 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:16.784 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:16.784 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:16.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:16.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:16.785 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:04:16.785 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:16.785 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:16.786 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:16.786 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:16.786 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:16.786 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:16.787 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:16.787 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:16.787 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:16.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:16.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:16.788 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:04:16.788 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:16.788 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:16.788 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:16.789 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:16.789 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:16.789 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:16.789 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:16.789 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:16.790 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:16.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:16.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:16.790 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:04:16.790 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:16.791 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:16.791 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:16.791 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:16.791 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:16.791 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:16.792 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:16.792 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:16.792 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:16.792 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:16.793 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:04:16.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:16.793 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:04:16.793 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:04:16.793 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:04:16.794 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:16.794 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:04:16.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:04:21.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:21.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:21.798 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:04:21.847 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:04:21.848 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:04:21.927 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:04:21.927 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:04:21.927 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:04:21.928 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:04:21.928 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:04:21.928 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:04:21.928 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:21.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:21.929 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:04:21.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:21.929 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:04:21.929 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:21.930 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:21.930 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:21.930 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:21.930 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:04:21.931 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:21.931 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:21.931 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:21.931 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:21.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:21.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:21.932 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:04:21.932 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:21.932 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:21.933 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:21.933 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:21.933 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:21.933 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:21.934 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:21.934 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:21.934 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:21.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:21.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:21.935 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:04:21.935 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:21.935 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:21.935 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:21.936 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:21.936 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:21.936 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:21.936 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:21.937 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:21.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:21.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:21.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:21.937 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:04:21.938 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:21.938 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:21.938 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:21.938 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:21.939 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:21.939 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:21.939 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:21.939 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:21.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:21.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:21.940 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:04:21.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:21.940 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:04:21.941 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:04:21.941 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:04:21.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:21.941 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:04:21.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:04:26.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:26.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:26.945 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:04:26.994 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:04:26.994 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:04:27.049 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:04:27.050 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:04:27.050 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:04:27.050 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:04:27.051 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:04:27.051 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:04:27.051 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:27.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:27.052 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:04:27.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:27.052 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:04:27.052 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:27.053 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:27.053 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:27.053 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:27.053 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:04:27.054 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:27.054 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:27.054 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:27.054 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:27.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:27.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:27.055 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:04:27.055 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:27.056 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:27.056 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:27.056 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:27.056 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:27.057 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:27.057 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:27.057 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:27.057 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:27.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:27.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:27.058 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:04:27.058 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:27.058 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:27.059 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:27.059 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:27.059 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:27.059 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:27.060 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:27.060 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:27.060 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:27.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:27.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:27.061 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:04:27.061 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:27.061 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:27.061 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:27.061 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:27.062 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:27.062 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:27.062 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:27.062 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:27.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:27.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:27.063 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:04:27.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:27.064 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:04:27.064 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:04:27.064 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:04:27.064 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:27.065 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:04:27.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:04:32.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:32.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:32.068 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:04:32.115 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:04:32.115 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:04:32.184 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:04:32.184 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:04:32.185 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:04:32.185 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:04:32.185 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:04:32.185 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:04:32.186 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:32.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:32.186 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:04:32.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:32.187 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:04:32.187 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:32.187 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:32.187 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:32.187 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:32.188 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:04:32.188 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:32.188 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:32.188 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:32.189 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:32.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:32.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:32.189 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:04:32.189 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:32.190 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:32.190 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:32.190 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:32.190 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:32.191 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:32.191 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:32.191 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:32.191 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:32.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:32.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:32.192 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:04:32.192 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:32.192 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:32.193 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:32.193 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:32.193 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:32.193 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:32.193 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:32.194 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:32.194 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:32.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:32.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:32.195 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:04:32.195 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:32.195 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:32.195 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:32.196 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:32.196 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:32.196 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:32.196 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:32.196 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:32.197 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:32.197 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:32.197 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:04:32.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:32.197 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:04:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:04:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:04:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:32.199 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:04:32.200 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:04:37.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:37.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:37.203 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:04:37.254 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:04:37.254 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:04:37.312 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:04:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:04:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:04:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:04:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:04:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:04:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:04:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:04:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:04:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:04:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:04:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:04:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:04:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:04:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:04:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:04:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:37.327 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:04:37.328 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:04:42.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:42.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:42.331 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:04:42.380 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:04:42.381 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:04:42.463 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:04:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:04:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:04:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:04:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:04:42.464 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:04:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:04:42.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:04:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:42.466 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:04:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:42.467 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:42.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:04:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:42.469 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:42.470 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:42.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:04:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:42.472 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:42.473 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:42.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:04:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:42.475 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:42.476 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:42.477 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:04:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:04:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:04:42.478 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:04:42.479 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:42.479 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:04:42.480 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:04:47.482 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:47.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:47.483 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:04:47.534 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:04:47.534 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:04:47.606 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:04:47.607 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:04:47.607 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:04:47.607 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:04:47.608 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:04:47.608 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:04:47.608 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:47.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:47.608 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:04:47.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:47.609 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:04:47.609 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:47.609 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:47.610 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:47.610 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:47.610 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:04:47.611 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:47.611 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:47.611 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:47.612 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:47.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:47.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:47.612 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:04:47.613 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:47.613 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:47.613 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:47.613 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:47.614 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:47.614 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:47.615 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:47.615 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:47.615 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:47.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:47.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:47.616 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:04:47.616 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:47.617 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:47.617 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:47.617 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:47.617 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:47.617 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:47.618 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:47.618 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:47.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:47.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:47.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:47.619 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:04:47.619 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:47.619 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:47.619 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:47.621 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:47.621 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:47.622 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:47.622 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:47.622 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:47.622 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:47.623 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:47.623 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:04:47.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:47.623 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:04:47.624 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:04:47.624 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:04:47.624 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:47.624 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:04:47.626 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:04:52.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:52.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:52.628 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:04:52.677 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:04:52.678 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:04:52.730 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:04:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:04:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:04:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:04:52.731 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:04:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:04:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:52.732 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:04:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:04:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:52.733 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:04:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:52.734 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:52.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:04:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:52.736 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:52.737 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:52.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:04:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:52.739 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:52.740 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:04:52.741 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:52.742 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:52.743 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:04:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:04:52.744 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:04:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:04:52.745 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:52.745 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:04:52.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:04:57.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:57.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:04:57.749 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:04:57.798 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:04:57.799 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:04:57.876 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:04:57.876 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:04:57.876 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:04:57.877 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:04:57.877 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:04:57.877 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:04:57.877 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:57.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:57.878 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:04:57.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:57.878 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:04:57.879 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:57.879 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:57.879 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:57.879 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:57.880 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:04:57.880 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:57.880 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:57.880 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:57.880 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:57.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:57.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:57.881 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:04:57.881 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:57.882 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:57.882 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:57.882 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:57.882 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:57.883 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:57.883 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:57.883 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:57.883 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:57.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:57.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:57.884 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:04:57.884 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:57.884 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:57.885 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:57.885 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:57.885 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:57.885 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:57.886 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:57.886 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:57.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:57.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:57.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:04:57.887 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:04:57.887 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:04:57.887 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:04:57.887 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:04:57.888 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:04:57.888 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:04:57.888 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:04:57.888 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:04:57.888 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:04:57.889 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:57.889 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:57.889 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:04:57.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:04:57.890 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:04:57.890 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:04:57.890 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:04:57.890 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:04:57.891 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:04:57.892 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:05:02.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:02.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:02.894 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:05:02.945 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:05:02.946 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:05:03.020 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:05:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:05:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:05:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:05:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:05:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:05:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:05:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:05:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:05:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:05:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:05:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:05:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:05:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:05:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:05:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:05:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:03.036 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:05:03.038 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:05:08.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:08.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:08.040 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:05:08.087 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:05:08.087 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:05:08.142 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:05:08.142 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:05:08.143 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:05:08.143 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:05:08.143 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:05:08.143 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:05:08.143 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:08.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:08.144 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:05:08.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:08.145 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:05:08.145 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:08.145 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:08.145 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:08.145 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:08.146 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:05:08.146 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:08.146 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:08.146 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:08.147 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:08.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:08.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:08.148 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:05:08.148 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:08.148 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:08.148 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:08.148 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:08.149 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:08.149 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:08.149 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:08.149 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:08.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:08.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:08.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:08.150 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:05:08.151 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:08.151 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:08.151 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:08.152 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:08.152 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:08.152 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:08.152 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:08.152 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:08.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:08.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:08.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:08.153 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:05:08.154 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:08.154 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:08.154 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:08.154 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:08.155 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:08.155 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:08.155 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:08.155 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:08.156 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:08.156 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:08.156 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:05:08.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:08.157 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:05:08.157 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:05:08.157 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:05:08.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:08.157 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:05:08.159 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:05:13.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:13.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:13.163 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:05:13.212 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:05:13.212 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:05:13.281 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:05:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:05:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:05:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:05:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:05:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:05:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:05:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:05:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:05:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:13.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:13.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:13.287 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:05:13.287 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:05:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:05:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:05:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:05:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:05:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:05:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:13.296 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:05:13.299 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:05:18.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:18.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:18.302 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:05:18.350 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:05:18.351 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:05:18.405 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:05:18.406 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:05:18.407 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:05:18.407 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:05:18.408 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:05:18.408 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:05:18.408 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:18.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:18.410 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:05:18.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:18.411 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:05:18.411 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:18.412 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:18.412 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:18.414 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:18.414 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:05:18.414 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:18.414 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:18.415 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:18.415 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:18.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:18.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:18.416 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:05:18.416 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:18.416 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:18.416 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:18.416 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:18.417 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:18.417 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:18.417 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:18.417 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:18.418 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:18.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:18.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:18.418 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:05:18.419 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:18.419 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:18.419 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:18.419 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:18.420 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:18.420 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:18.420 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:18.420 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:18.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:18.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:18.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:18.421 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:05:18.421 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:18.422 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:18.422 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:18.422 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:18.423 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:18.423 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:18.423 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:18.423 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:18.424 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:18.424 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:18.424 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:05:18.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:18.424 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:05:18.425 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:05:18.425 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:05:18.425 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:18.425 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:05:18.427 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:05:23.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:23.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:23.430 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:05:23.480 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:05:23.480 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:05:23.557 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:05:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:05:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:05:23.558 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:05:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:05:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:05:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:23.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:05:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:05:23.560 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:23.561 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:05:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:23.562 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:23.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:05:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:23.564 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:23.565 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:05:23.567 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:23.568 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:23.569 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:23.570 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:23.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:23.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:05:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:23.573 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:23.574 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:23.575 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:23.576 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:05:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:05:23.577 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:05:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:05:23.578 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:23.578 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:05:23.580 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:05:28.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:28.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:28.582 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:05:28.633 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:05:28.633 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:05:28.715 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:05:28.715 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:05:28.715 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:05:28.716 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:05:28.716 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:05:28.716 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:05:28.716 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:28.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:28.717 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:05:28.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:28.717 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:05:28.718 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:28.718 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:28.718 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:28.718 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:28.719 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:05:28.719 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:28.719 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:28.719 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:28.720 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:28.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:28.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:28.720 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:05:28.721 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:28.721 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:28.721 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:28.721 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:28.722 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:28.722 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:28.722 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:28.722 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:28.723 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:28.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:28.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:28.723 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:05:28.723 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:28.724 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:28.724 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:28.724 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:28.724 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:28.725 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:28.725 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:28.725 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:28.725 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:28.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:28.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:28.726 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:05:28.726 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:28.726 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:28.727 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:28.727 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:28.727 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:28.727 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:28.728 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:28.728 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:28.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:28.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:28.729 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:05:28.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:28.729 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:05:28.729 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:05:28.729 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:05:28.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:28.730 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:05:28.731 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:05:33.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:33.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:33.733 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:05:33.783 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:05:33.783 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:05:33.854 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:05:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:05:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:05:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:05:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:05:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:05:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:05:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:05:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:05:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:05:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:05:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:05:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:05:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:05:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:05:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:05:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:33.869 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:05:33.870 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:05:38.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:38.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:38.872 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:05:38.922 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:05:38.922 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:05:39.006 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:05:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:05:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:05:39.007 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:05:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:05:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:05:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:39.008 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:05:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:05:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:39.009 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:39.010 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:05:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:39.011 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:39.012 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:05:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:39.013 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:39.014 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:39.015 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:05:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:39.016 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:39.017 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:39.018 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:05:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:39.019 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:39.020 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:39.021 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:05:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:05:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:05:39.022 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:05:39.023 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:39.023 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:05:39.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:05:44.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:44.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:44.026 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:05:44.074 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:05:44.075 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:05:44.138 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:05:44.139 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:05:44.139 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:05:44.139 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:05:44.140 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:05:44.140 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:05:44.141 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:44.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:44.142 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:05:44.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:44.143 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:05:44.144 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:44.144 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:44.145 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:44.145 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:44.146 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:05:44.147 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:44.148 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:44.148 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:44.148 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:44.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:44.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:44.149 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:05:44.150 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:44.150 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:44.150 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:44.150 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:44.151 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:44.151 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:44.151 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:44.151 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:44.152 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:44.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:44.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:44.152 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:05:44.153 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:44.153 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:44.153 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:44.153 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:44.154 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:44.154 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:44.155 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:44.155 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:44.155 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:44.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:44.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:44.156 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:05:44.157 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:44.157 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:44.158 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:44.159 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:44.159 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:44.160 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:44.160 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:44.160 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:44.160 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:44.160 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:44.161 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:05:44.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:44.161 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:05:44.161 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:05:44.162 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:05:44.162 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:44.162 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:05:44.164 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:05:49.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:49.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:49.166 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:05:49.216 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:05:49.216 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:05:49.304 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:05:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:05:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:05:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:05:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:05:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:05:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:05:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:05:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:05:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:05:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:05:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:05:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:49.317 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:49.317 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:49.317 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:49.317 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:49.317 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:05:49.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:49.318 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:05:49.318 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:05:49.318 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:05:49.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:49.319 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:05:49.320 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:05:54.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:54.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:54.323 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:05:54.372 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:05:54.372 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:05:54.453 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:05:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:05:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:05:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:05:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:05:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:05:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:05:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:05:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:05:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:54.458 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:54.458 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:54.458 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:54.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:54.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:54.459 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:05:54.459 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:54.459 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:54.460 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:54.460 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:54.460 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:54.460 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:54.461 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:54.461 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:54.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:54.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:54.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:05:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:05:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:05:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:05:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:05:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:05:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:54.468 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:05:54.469 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:05:59.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:59.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:05:59.472 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:05:59.522 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:05:59.522 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:05:59.577 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:05:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:05:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:05:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:05:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:05:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:05:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:05:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:05:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:05:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:05:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:05:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:05:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:05:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:05:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:05:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:05:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:05:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:05:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:05:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:05:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:05:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:05:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:05:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:05:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:05:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:05:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:05:59.592 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:05:59.593 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:06:04.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:04.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:04.595 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:04.646 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:04.646 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:04.727 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:04.727 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:04.728 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:04.728 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:04.728 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:04.728 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:04.729 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:04.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:04.729 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:04.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:04.729 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:04.730 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:04.730 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:04.730 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:04.730 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:04.731 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:04.731 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:04.731 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:04.731 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:04.731 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:04.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:04.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:04.732 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:04.732 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:04.733 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:04.733 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:04.733 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:04.734 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:04.734 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:04.734 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:04.734 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:04.734 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:04.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:04.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:04.735 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:04.735 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:04.737 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:04.737 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:04.737 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:04.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:04.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:04.741 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:04.743 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:06:09.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:09.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:09.747 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:09.796 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:09.796 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:09.870 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:09.870 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:09.870 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:09.871 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:09.871 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:09.871 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:09.871 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:09.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:09.872 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:09.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:09.872 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:09.872 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:09.873 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:09.873 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:09.873 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:09.873 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:09.874 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:09.874 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:09.874 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:09.874 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:09.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:09.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:09.875 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:09.875 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:09.875 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:09.876 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:09.876 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:09.876 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:09.876 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:09.876 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:09.877 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:09.877 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:09.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:09.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:09.877 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:09.878 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:09.878 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:09.878 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:09.878 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:09.879 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:09.879 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:09.879 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:09.879 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:09.880 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:09.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:09.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:09.880 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:09.880 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:09.881 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:09.881 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:09.881 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:09.881 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:09.881 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:09.882 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:09.882 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:09.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:09.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:09.883 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:09.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:09.883 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:09.883 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:09.883 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:09.884 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:09.884 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:09.885 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641909851.416936_36.1 target= 2022-01-11T14:06:14.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:14.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:14.887 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:14.936 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:14.936 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:15.004 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:15.005 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:15.005 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:15.006 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:15.006 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:15.007 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:15.007 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.008 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:15.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:15.008 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:15.008 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:15.009 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:15.009 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:15.009 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:15.009 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:15.009 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:15.010 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:15.010 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.011 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:15.011 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:15.011 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:15.012 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:15.012 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:15.012 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:15.012 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:15.013 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:15.013 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.014 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.014 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:15.015 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:15.016 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:15.016 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:15.017 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:15.018 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:15.018 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:15.019 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:15.019 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.020 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.020 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:15.022 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:15.022 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:15.023 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:15.023 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:15.024 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:15.024 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:15.025 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:15.025 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.025 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.026 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.026 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.026 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:15.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.027 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:15.027 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:15.027 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:15.028 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.028 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:15.029 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:06:15.030 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:06:15.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:15.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:15.032 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:15.082 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:15.083 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:15.135 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:06:15.135 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:06:15.136 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:06:15.136 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:06:15.136 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:06:15.136 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:15.137 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:15.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:15.137 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:06:15.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:15.138 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:06:15.138 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:15.138 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:15.138 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:15.139 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:15.139 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:06:15.139 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:15.139 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:15.140 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:15.140 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:15.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:15.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:15.140 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:06:15.141 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:15.141 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:15.141 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:15.141 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:15.142 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:15.142 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:15.142 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.142 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:15.143 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:15.143 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:15.143 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:15.144 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:06:15.144 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:15.144 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:15.144 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:15.145 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:15.145 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:15.146 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:15.146 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.147 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:15.147 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:15.147 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:15.148 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:15.148 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:06:15.148 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:15.148 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:15.149 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:15.149 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:15.149 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:15.149 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:15.150 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.150 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:15.150 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:15.150 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:15.151 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:06:15.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:15.151 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:06:15.152 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:06:15.152 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:06:15.152 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:15.152 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:06:15.153 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:06:15.154 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:06:15.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:15.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:15.155 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:15.201 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:15.202 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:15.270 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:15.473 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:15.474 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:15.475 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:15.476 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:15.477 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:15.478 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.478 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:15.479 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:15.480 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:15.481 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:15.482 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:15.483 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:15.486 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:15.486 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:15.487 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:15.488 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.489 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:15.491 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:15.492 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:15.493 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.494 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:15.496 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:15.497 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:15.498 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:15.498 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.499 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.500 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:15.501 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:15.502 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:15.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.502 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:15.503 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:06:15.504 INFO:rgw_multi.tests:meta checkpoint 2022-01-11T14:06:15.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:15.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:15.506 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:15.579 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:15.581 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:15.623 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:06:15.637 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:06:15.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:15.638 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:15.639 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:04:11.416936Z" 2022-01-11T14:06:15.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:15.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:15.640 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:15.640 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:06:15.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:15.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:15.641 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:15.641 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:06:15.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:15.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:15.643 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:15.643 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:06:15.643 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:15.644 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:06:15.644 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:06:15.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:15.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:15.647 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:15.700 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:15.701 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:15.752 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:15.752 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:15.753 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:15.754 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:15.754 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:15.756 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:15.757 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:15.758 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:15.759 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:15.760 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:15.762 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:15.763 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:15.764 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:15.765 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:15.769 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:15.770 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.771 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.772 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:15.773 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:15.774 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:15.775 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.776 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:15.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:15.779 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:15.780 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:15.780 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:15.783 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:06:15.783 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:06:15.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:15.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:15.785 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:15.834 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:15.835 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:15.895 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:06:15.896 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:06:15.896 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:06:15.896 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:06:15.898 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:06:15.898 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:15.899 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:15.900 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:15.900 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:06:15.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:15.901 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:06:15.901 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:15.901 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:15.902 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:15.902 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:15.902 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:06:15.903 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:15.903 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:15.904 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:15.904 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:15.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:15.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:15.905 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:06:15.906 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:15.907 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:15.909 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:15.909 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:15.910 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:15.910 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:15.910 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.910 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:15.911 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:15.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:15.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:15.911 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:06:15.911 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:15.912 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:15.912 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:15.912 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:15.912 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:15.914 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:15.915 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.915 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:15.916 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:15.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:15.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:15.918 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:06:15.919 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:15.919 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:15.920 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:15.920 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:15.920 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:15.920 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:15.921 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:15.921 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:15.921 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:15.921 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:15.922 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:06:15.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:15.922 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:06:15.923 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:06:15.923 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:06:15.923 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:15.923 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:06:15.925 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:06:15.925 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:06:15.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:15.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:15.926 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:15.975 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:15.975 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:16.029 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:16.029 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:16.030 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:16.031 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:16.034 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:16.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:16.036 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:16.037 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:16.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:16.039 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:16.040 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:16.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:16.042 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:16.043 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:16.044 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:16.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:16.046 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:16.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:16.047 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:16.048 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:06:16.049 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_17f142526478a11e8f62d3933be552d546c4fd19/qa/../src/test/rgw/rgw_multi' with args=['tests.py', 'tests_ps.py'] 2022-01-11T14:06:16.102 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-1 2022-01-11T14:06:19.394 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-2 2022-01-11T14:06:22.941 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-3 2022-01-11T14:06:23.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:23.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:23.009 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:23.053 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:23.053 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:23.091 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:06:23.103 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:06:23.103 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:23.104 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:23.104 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:04:11.416936Z" 2022-01-11T14:06:23.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:23.105 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:23.105 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909982.991799_51.1", 2022-01-11T14:06:23.105 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:22.991799Z" 2022-01-11T14:06:23.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:23.106 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:23.106 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:23.107 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:06:23.107 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:23.107 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:23.107 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:23.108 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:06:23.108 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:23.108 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:06:23.109 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:06:23.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:23.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:23.111 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:23.157 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:23.158 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:23.224 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909979.380783_40.1", 2022-01-11T14:06:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:19.380783Z", 2022-01-11T14:06:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:23.236 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:23.236 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:23.236 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:23.236 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:23.239 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:23.240 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1641909982.991799_51.1 target=1_1641909979.380783_40.1 2022-01-11T14:06:28.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:28.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:28.243 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:28.292 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:28.293 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:28.371 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909982.991799_51.1", 2022-01-11T14:06:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:22.991799Z", 2022-01-11T14:06:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:28.399 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:28.407 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:06:28.408 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:06:28.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:28.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:28.409 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:28.458 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:28.459 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:28.525 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:06:28.526 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:06:28.526 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:06:28.526 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:06:28.526 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:06:28.527 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:28.527 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:28.527 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:28.527 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:06:28.527 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:28.528 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:06:28.528 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:28.528 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:28.528 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:28.529 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:28.529 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:06:28.529 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:28.529 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:28.529 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:28.530 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:28.530 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:28.530 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:28.533 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:06:28.533 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:28.533 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:28.534 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909982.991799_51.1", 2022-01-11T14:06:28.534 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:28.534 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:28.534 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:28.535 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:22.991799Z", 2022-01-11T14:06:28.535 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:28.535 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:28.535 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:28.535 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:28.536 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:06:28.536 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:28.536 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:28.536 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:28.537 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:28.537 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:28.537 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:28.537 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:28.537 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:28.538 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:28.538 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:28.538 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:28.538 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:06:28.539 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:28.539 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:28.539 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:28.539 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:28.539 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:28.540 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:28.540 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:28.540 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:28.540 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:28.541 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:28.541 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:06:28.541 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:28.541 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:06:28.541 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:06:28.542 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:06:28.542 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:28.542 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:06:28.543 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:06:28.544 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:06:28.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:28.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:28.546 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:28.591 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:28.591 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:28.639 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:28.640 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:28.640 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:28.640 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:28.641 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:28.641 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:28.641 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:28.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:28.642 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:28.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:28.642 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:28.642 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:28.643 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:28.643 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:28.643 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:28.644 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:28.644 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:28.644 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:28.644 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:28.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:28.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:28.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:28.645 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:28.646 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:28.646 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:28.646 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909982.991799_51.1", 2022-01-11T14:06:28.646 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:28.647 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:28.648 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:28.648 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:22.991799Z", 2022-01-11T14:06:28.648 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:28.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:28.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:28.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:28.649 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:28.649 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:28.650 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:28.650 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:28.650 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:28.650 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:28.651 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:28.651 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:28.651 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:28.651 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:28.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:28.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:28.652 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:28.652 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:28.652 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:28.653 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:28.653 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:28.653 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:28.654 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:28.654 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:28.654 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:28.654 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:28.654 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:28.655 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:28.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:28.655 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:28.655 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:28.656 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:28.656 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:28.656 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:28.658 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:06:28.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2022-01-11T14:06:28.688 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-4 2022-01-11T14:06:28.702 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-5 2022-01-11T14:06:28.728 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-6 2022-01-11T14:06:28.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:28.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:28.760 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:28.804 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:28.805 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:28.864 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:06:28.884 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:06:28.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:28.884 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:28.884 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:04:11.416936Z" 2022-01-11T14:06:28.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:28.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:28.885 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909988.748737_63.1", 2022-01-11T14:06:28.885 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:28.748737Z" 2022-01-11T14:06:28.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:28.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:28.886 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:28.886 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:06:28.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:28.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:28.887 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:28.888 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:06:28.888 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:28.888 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:06:28.890 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:06:28.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:28.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:28.891 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:28.936 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:28.937 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:29.005 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:29.006 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:29.006 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:29.006 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:29.006 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:29.007 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:29.007 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:29.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:29.007 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:29.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:29.008 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:29.008 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:29.008 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:29.009 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:29.009 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:29.009 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:29.010 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:29.010 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:29.010 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:29.010 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:29.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:29.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:29.011 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:29.011 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:29.012 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:29.012 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909988.697851_52.1", 2022-01-11T14:06:29.012 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:29.012 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:29.012 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:29.013 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:28.697851Z", 2022-01-11T14:06:29.013 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:29.013 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:29.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:29.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:29.014 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:29.014 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:29.014 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:29.014 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:29.015 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:29.015 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:29.015 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:29.015 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:29.016 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:29.016 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:29.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:29.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:29.016 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:29.017 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:29.017 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:29.017 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:29.017 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:29.018 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:29.018 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:29.018 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:29.018 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:29.018 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:29.019 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:29.019 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:29.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:29.019 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:29.020 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:29.020 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:29.020 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:29.020 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:29.022 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1641909988.748737_63.1 target=1_1641909988.697851_52.1 2022-01-11T14:06:34.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:34.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:34.026 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:34.074 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:34.075 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:34.157 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:34.158 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:34.159 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:34.160 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:34.161 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:34.162 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:34.162 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:34.162 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:34.163 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:34.163 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:34.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:34.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:34.164 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:34.164 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:34.164 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:34.164 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909988.748737_63.1", 2022-01-11T14:06:34.165 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:34.165 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:34.165 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:34.165 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:28.748737Z", 2022-01-11T14:06:34.166 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:34.166 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:34.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:34.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:34.167 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:34.167 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:34.167 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:34.169 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:34.169 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:34.169 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:34.170 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:34.170 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:34.170 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:34.171 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:34.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:34.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:34.171 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:34.171 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:34.172 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:34.172 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:34.172 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:34.172 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:34.173 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:34.173 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:34.173 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:34.173 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:34.174 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:34.174 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:34.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:34.174 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:34.174 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:34.175 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:34.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:34.175 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:34.176 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:06:34.177 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:06:34.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:34.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:34.179 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:34.228 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:34.228 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:34.283 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:06:34.284 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:06:34.284 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:06:34.285 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:06:34.285 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:06:34.286 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:34.287 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:34.287 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:34.287 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:06:34.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:34.288 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:06:34.289 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:34.289 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:34.290 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:34.290 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:34.290 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:06:34.291 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:34.292 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:34.292 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:34.293 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:34.293 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:34.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:34.294 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:06:34.294 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:34.295 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:34.295 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909988.743778_61.1", 2022-01-11T14:06:34.296 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:34.296 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:34.297 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:34.297 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:28.743778Z", 2022-01-11T14:06:34.297 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:34.298 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:34.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:34.299 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:34.299 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:06:34.300 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:34.300 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:34.300 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:34.301 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:34.302 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:34.303 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:34.303 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:34.303 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:34.304 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:34.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:34.305 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:34.305 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:06:34.306 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:34.306 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:34.307 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:34.307 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:34.308 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:34.308 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:34.308 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:34.309 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:34.310 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:34.310 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:34.311 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:06:34.311 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:34.311 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:06:34.312 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:06:34.313 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:06:34.313 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:34.313 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:06:34.315 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1641909988.748737_63.1 target=1_1641909988.743778_61.1 2022-01-11T14:06:39.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:39.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:39.320 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:39.371 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:39.372 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:39.447 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:06:39.447 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:06:39.448 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:06:39.448 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:06:39.449 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:06:39.449 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:39.449 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:39.450 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:39.450 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:06:39.451 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:39.451 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:06:39.452 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:39.452 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:39.452 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:39.453 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:39.454 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:06:39.454 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:39.454 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:39.455 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:39.455 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:39.455 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:39.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:39.455 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:06:39.456 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:39.456 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:39.456 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909988.743778_61.1", 2022-01-11T14:06:39.456 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:39.457 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:39.457 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:39.457 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:28.743778Z", 2022-01-11T14:06:39.457 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:39.457 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:39.458 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:39.458 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:39.458 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:06:39.458 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:39.458 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:39.459 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:39.459 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:39.459 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:39.459 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:39.460 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:39.460 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:39.460 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:39.460 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:39.461 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:39.461 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:06:39.461 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:39.461 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:39.462 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:39.462 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:39.462 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:39.462 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:39.463 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:39.463 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:39.463 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:39.463 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:39.463 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:06:39.464 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:39.464 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:06:39.464 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:06:39.464 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:06:39.465 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:39.465 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:06:39.466 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1641909988.748737_63.1 target=1_1641909988.743778_61.1 2022-01-11T14:06:44.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:44.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:44.469 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:44.517 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:44.518 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:44.590 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:06:44.590 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:06:44.591 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:06:44.591 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:06:44.591 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:06:44.591 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:44.592 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:44.592 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:44.592 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:06:44.592 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:44.592 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:06:44.593 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:44.593 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:44.593 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:44.593 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:44.593 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:06:44.594 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:44.594 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:44.594 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:44.594 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:44.595 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:44.595 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:44.595 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:06:44.595 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:44.596 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:44.596 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909988.743778_61.1", 2022-01-11T14:06:44.596 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:44.596 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:44.596 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:44.597 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:28.743778Z", 2022-01-11T14:06:44.597 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:44.597 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:44.597 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:44.598 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:44.598 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:06:44.598 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:44.598 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:44.598 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:44.599 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:44.599 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:44.599 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:44.599 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:44.600 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:44.600 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:44.600 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:44.600 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:44.601 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:06:44.601 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:44.601 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:44.601 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:44.601 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:44.602 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:44.602 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:44.602 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:44.602 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:44.603 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:44.603 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:44.603 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:06:44.603 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:44.604 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:06:44.604 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:06:44.604 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:06:44.604 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:44.604 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:06:44.606 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1641909988.748737_63.1 target=1_1641909988.743778_61.1 2022-01-11T14:06:49.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:49.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:49.609 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:49.659 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:49.660 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:49.737 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:06:49.738 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:06:49.738 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:06:49.739 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:06:49.740 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:06:49.740 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:49.740 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:49.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:49.741 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:06:49.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:49.742 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:06:49.742 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:49.743 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:49.743 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:49.744 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:49.744 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:06:49.744 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:49.744 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:49.745 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:49.745 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:49.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:49.746 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:49.746 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:06:49.747 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:49.747 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:49.748 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909988.748737_63.1", 2022-01-11T14:06:49.748 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:49.748 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:49.749 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:49.749 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:28.748737Z", 2022-01-11T14:06:49.749 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:49.750 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:49.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:49.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:49.751 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:06:49.751 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:49.752 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:49.752 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:49.752 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:49.753 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:49.753 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:49.754 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:49.754 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:49.754 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:49.755 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:49.755 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:49.756 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:06:49.756 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:49.756 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:49.756 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:49.757 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:49.757 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:49.757 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:49.758 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:49.758 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:49.758 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:49.758 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:06:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:06:49.760 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:06:49.760 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:06:49.761 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:49.761 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:06:49.762 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:06:49.763 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:06:49.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:49.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:49.765 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:49.816 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:49.817 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:49.875 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:49.877 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:49.878 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:49.879 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:49.880 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:49.881 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:49.882 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:49.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909988.748737_63.1", 2022-01-11T14:06:49.884 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:28.748737Z", 2022-01-11T14:06:49.885 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:49.886 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:49.887 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:49.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:49.889 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:49.890 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:49.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:49.892 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:49.893 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:49.894 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:49.895 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:06:50.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:50.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:50.174 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:50.215 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:50.216 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:50.252 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:06:50.265 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:06:50.265 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:50.266 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:50.266 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:04:11.416936Z" 2022-01-11T14:06:50.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:50.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:50.267 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909988.748737_63.1", 2022-01-11T14:06:50.267 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:28.748737Z" 2022-01-11T14:06:50.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:50.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:50.268 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:50.268 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:06:50.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:50.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:50.269 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:50.269 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:06:50.270 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:50.270 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:06:50.271 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:06:50.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:50.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:50.273 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:50.318 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:50.319 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:50.380 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:50.380 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:50.381 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:50.381 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:50.381 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:50.382 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:50.382 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:50.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:50.383 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:50.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:50.383 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:50.383 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:50.384 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:50.384 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:50.384 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:50.385 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:50.385 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:50.385 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:50.385 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:50.386 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:50.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:50.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:50.386 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:50.386 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:50.387 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:50.387 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909988.748737_63.1", 2022-01-11T14:06:50.388 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:50.388 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:50.388 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:50.389 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:28.748737Z", 2022-01-11T14:06:50.389 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:50.390 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:50.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:50.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:50.391 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:50.391 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:50.392 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:50.392 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:50.393 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:50.393 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:50.394 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:50.394 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:50.395 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:50.395 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:50.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:50.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:50.396 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:50.397 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:50.397 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:50.397 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:50.398 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:50.398 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:50.399 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:50.399 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:50.400 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:50.400 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:50.401 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:50.401 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:50.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:50.402 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:50.402 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:50.403 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:50.404 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:50.404 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:50.406 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:06:50.407 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:06:50.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:50.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:50.411 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:50.454 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:50.455 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:50.524 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:06:50.525 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:06:50.525 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:06:50.525 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:06:50.526 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:06:50.526 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:50.526 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:50.527 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:50.527 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:06:50.527 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:50.527 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:06:50.528 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:50.528 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:50.528 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:50.528 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:50.528 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:06:50.529 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:50.529 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:50.529 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:50.529 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:50.530 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:50.530 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:50.530 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:06:50.530 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:50.531 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:50.531 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909988.748737_63.1", 2022-01-11T14:06:50.531 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:50.531 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:50.531 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:50.532 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:28.748737Z", 2022-01-11T14:06:50.532 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:50.532 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:50.532 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:50.533 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:50.533 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:06:50.533 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:50.533 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:50.533 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:50.534 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:50.534 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:50.534 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:50.534 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:50.535 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:50.535 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:50.535 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:50.535 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:50.536 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:06:50.536 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:50.536 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:50.536 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:50.537 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:50.537 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:50.537 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:50.537 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:50.537 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:50.538 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:50.538 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:50.538 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:06:50.538 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:50.539 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:06:50.539 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:06:50.539 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:06:50.539 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:50.540 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:06:50.541 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:06:50.542 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:06:50.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:50.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:50.543 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:50.586 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:50.587 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:50.660 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:50.660 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:50.660 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:50.661 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:50.661 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:50.661 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909988.748737_63.1", 2022-01-11T14:06:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:28.748737Z", 2022-01-11T14:06:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:50.672 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:50.672 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:50.672 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:50.672 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:50.673 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:50.673 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:50.673 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:50.673 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:50.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:50.675 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:50.676 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:06:50.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2022-01-11T14:06:50.702 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-7 2022-01-11T14:06:50.727 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-8 2022-01-11T14:06:50.762 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-9 2022-01-11T14:06:50.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:50.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:50.802 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:50.845 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:50.846 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:50.883 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:06:50.895 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:06:50.895 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:50.895 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:50.896 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:04:11.416936Z" 2022-01-11T14:06:50.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:50.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:50.896 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910010.786119_75.1", 2022-01-11T14:06:50.896 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:50.786119Z" 2022-01-11T14:06:50.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:50.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:50.898 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:50.899 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:06:50.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:50.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:50.900 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:50.900 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:06:50.901 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:50.901 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:06:50.903 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:06:50.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:50.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:50.906 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:50.949 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:50.949 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:51.016 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:51.017 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:51.017 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:51.018 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:51.018 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:51.018 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:51.018 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:51.020 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:51.020 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:51.020 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:51.020 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:51.021 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:51.021 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:51.021 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:51.022 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:51.022 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:51.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:51.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:51.023 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:51.023 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:51.023 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:51.023 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910010.715248_64.1", 2022-01-11T14:06:51.024 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:51.024 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:51.024 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:51.024 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:50.715248Z", 2022-01-11T14:06:51.024 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:51.025 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:51.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:51.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:51.025 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:51.026 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:51.026 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:51.026 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:51.026 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:51.026 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:51.027 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:51.027 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:51.027 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:51.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:51.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:51.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:51.028 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:51.028 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:51.029 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:51.030 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:51.030 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:51.030 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:51.030 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:51.031 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:51.031 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:51.031 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:51.031 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:51.032 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:51.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:51.032 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:51.033 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:51.033 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:51.033 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:51.033 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:51.035 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1641910010.786119_75.1 target=1_1641910010.715248_64.1 2022-01-11T14:06:56.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:56.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:56.038 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:56.083 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:56.084 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:56.167 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:56.167 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.167 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:56.168 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:56.168 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:56.168 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:56.168 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910010.786119_75.1", 2022-01-11T14:06:56.169 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:56.169 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:56.169 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:56.169 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:50.786119Z", 2022-01-11T14:06:56.170 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.170 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:56.171 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:56.171 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:56.171 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:56.171 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:56.172 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:56.172 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:56.172 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:56.172 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:56.172 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.173 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:56.173 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:56.174 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:56.174 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:56.174 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:56.174 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:56.175 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:56.175 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:56.175 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:56.175 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.176 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.176 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:56.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.176 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:56.177 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:56.177 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:56.177 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.177 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:56.178 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:06:56.179 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:06:56.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:56.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:56.180 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:56.229 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:56.229 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:56.307 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:06:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:06:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:06:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:06:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:06:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:06:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:06:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:06:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:56.313 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:56.313 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:56.313 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:56.313 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:56.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:56.314 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:06:56.315 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:56.315 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:56.315 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910010.786119_75.1", 2022-01-11T14:06:56.315 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:56.316 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:56.316 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:56.316 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:50.786119Z", 2022-01-11T14:06:56.316 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:56.316 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:56.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:56.317 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:56.317 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:06:56.317 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:56.318 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:56.318 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:56.318 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:56.318 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:56.319 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:56.319 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:56.319 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:56.319 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:56.320 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:56.320 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:56.320 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:06:56.320 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:06:56.320 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:06:56.321 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:56.321 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:06:56.321 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:06:56.321 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:06:56.322 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:06:56.322 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:06:56.322 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:56.322 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:56.323 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:06:56.323 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:56.323 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:06:56.323 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:06:56.324 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:06:56.324 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:56.325 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:06:56.327 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:06:56.328 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:06:56.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:56.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:56.330 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:56.376 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:56.376 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:56.447 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:56.447 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:56.447 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:56.448 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:56.448 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:56.449 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:56.449 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.449 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:56.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:56.450 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:56.450 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:56.450 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:56.450 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:56.451 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:56.451 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:56.451 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:56.451 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:56.452 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:56.452 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:56.453 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:56.453 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:56.453 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910010.786119_75.1", 2022-01-11T14:06:56.453 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:56.454 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:56.454 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:56.454 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:50.786119Z", 2022-01-11T14:06:56.454 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.455 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:56.455 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:56.455 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:56.456 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:56.456 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:56.456 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:56.456 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:56.457 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:56.457 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:56.457 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:56.458 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:56.458 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:56.459 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:56.459 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:56.459 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:56.459 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:56.460 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:56.460 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:56.460 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.460 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.461 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:56.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.461 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:56.461 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:56.462 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:56.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.462 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:56.463 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:06:56.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:56.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:56.615 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:56.657 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:56.657 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:56.709 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:06:56.728 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:06:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:04:11.416936Z" 2022-01-11T14:06:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:06:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:06:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:06:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:06:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:06:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:06:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:06:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:06:56.732 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:06:56.733 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:06:56.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:56.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:06:56.736 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:06:56.778 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:06:56.778 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:06:56.853 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:06:56.853 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:06:56.854 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:06:56.854 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:06:56.854 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:06:56.854 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:06:56.855 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.855 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:06:56.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:56.855 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:06:56.856 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:56.856 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:56.856 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:06:56.856 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:56.857 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:06:56.857 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:56.857 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:06:56.857 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.857 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:56.858 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:06:56.859 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:56.859 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:56.859 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.544181_79.1", 2022-01-11T14:06:56.859 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:56.859 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:56.860 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:56.861 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.544181Z", 2022-01-11T14:06:56.861 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.861 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:56.862 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:06:56.862 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:56.863 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:56.863 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:56.863 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:56.863 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:56.864 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:56.864 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:56.864 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.864 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:06:56.865 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:06:56.865 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:06:56.865 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:06:56.866 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:06:56.866 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:06:56.866 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:06:56.866 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:06:56.866 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:06:56.867 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:06:56.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.867 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:06:56.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:06:56.868 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:06:56.868 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:06:56.868 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:06:56.869 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:06:56.869 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:06:56.871 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1641910016.595927_81.1 target=1_1641910016.544181_79.1 2022-01-11T14:07:01.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:01.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:01.873 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:07:01.924 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:07:01.924 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:07:01.992 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:07:01.992 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:07:01.993 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:07:01.993 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:07:01.993 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:07:01.993 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:07:01.994 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:01.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:01.994 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:07:01.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:01.995 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:07:01.996 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:01.996 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:01.996 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:07:01.996 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:01.997 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:07:01.997 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:01.997 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:07:01.997 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:01.998 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:01.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:01.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:01.998 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:07:01.998 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:01.999 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:01.999 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:07:01.999 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:01.999 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:02.000 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:02.000 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:07:02.000 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:02.000 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:02.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:02.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:02.001 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:07:02.001 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:02.002 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:02.002 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:07:02.002 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:02.003 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:02.003 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:02.003 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:07:02.003 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:02.003 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:02.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:02.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:02.004 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:07:02.004 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:02.005 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:02.005 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:07:02.005 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:02.005 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:02.005 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:02.006 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:07:02.006 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:02.006 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:02.006 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:07:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:07:02.007 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:07:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:07:02.008 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:02.008 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:07:02.009 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:07:02.010 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:07:02.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:02.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:02.011 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:07:02.062 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:07:02.062 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:07:02.129 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:07:02.130 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:07:02.130 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:07:02.130 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:07:02.130 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:07:02.131 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:07:02.131 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:07:02.131 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:07:02.132 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:07:02.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:07:02.132 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:07:02.132 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:07:02.132 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:07:02.133 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:07:02.133 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:07:02.133 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:07:02.133 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:07:02.134 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:07:02.134 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:07:02.134 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:07:02.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:07:02.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:07:02.135 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:07:02.135 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:07:02.135 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:07:02.135 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:07:02.136 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:07:02.136 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:07:02.136 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:07:02.136 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:07:02.137 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:07:02.137 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:07:02.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:07:02.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:07:02.137 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:07:02.138 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:07:02.138 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:07:02.138 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:07:02.138 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:07:02.139 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:07:02.139 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:07:02.139 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:07:02.139 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:07:02.139 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:07:02.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:07:02.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:07:02.140 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:07:02.140 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:07:02.140 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:07:02.141 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:07:02.141 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:07:02.141 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:07:02.141 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:07:02.142 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:07:02.142 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:07:02.142 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:07:02.142 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:07:02.142 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:07:02.143 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:07:02.143 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:07:02.143 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:07:02.143 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:07:02.144 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:07:02.144 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:07:02.145 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:07:02.146 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:07:02.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:02.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:02.147 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:07:02.191 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:07:02.191 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:07:02.241 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:07:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:07:02.241 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:07:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:07:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:07:02.242 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:07:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:02.243 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:07:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:07:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:02.244 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641909851.416936_36.1", 2022-01-11T14:07:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:07:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:02.245 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:04:11.416936Z", 2022-01-11T14:07:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:02.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:07:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:02.247 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:07:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:07:02.248 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:02.249 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:07:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:07:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:02.250 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:02.251 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:07:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:02.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:07:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:02.253 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:07:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:02.254 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:07:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:07:02.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:07:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:07:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:07:02.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:02.257 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:07:02.258 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:07:02.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2022-01-11T14:07:02.270 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-10 2022-01-11T14:07:02.288 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-11 2022-01-11T14:07:02.317 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-12 2022-01-11T14:07:02.389 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:07:02.391+0000 7fc1f48f2700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:02.390 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2022-01-11T14:07:02.391+0000 7f98c683c700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:02.390 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:07:02.391+0000 7f3d9dc06700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:02.589 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2022-01-11T14:07:02.590+0000 7f98c2834700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:02.590 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2022-01-11T14:07:02.590+0000 7f98c4838700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:02.590 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2022-01-11T14:07:02.590+0000 7f98c2834700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:02.591 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:07:02.591+0000 7f3d973f9700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:02.592 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:07:02.591+0000 7f3d92bf0700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:02.592 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:07:02.591+0000 7f3d95bf6700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:02.592 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:07:02.593+0000 7f3d98bfc700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:02.593 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:07:02.593+0000 7f3d99bfe700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:02.593 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:07:02.591+0000 7fc1f18ec700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:02.593 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:07:02.591+0000 7fc1fa0fd700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:02.594 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:07:02.591+0000 7fc1f18ec700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:02.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:02.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:02.957 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:07:03.008 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:07:03.009 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:07:03.096 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:07:03.135 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:07:03.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:07:03.135 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910022.347485_48.1", 2022-01-11T14:07:03.136 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:07:02.347485Z" 2022-01-11T14:07:03.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:07:03.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:07:03.136 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:07:03.137 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:07:03.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:07:03.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:07:03.137 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:07:03.137 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:07:03.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:07:03.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:07:03.139 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:07:03.139 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:07:03.139 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:07:03.139 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:07:03.141 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:07:03.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:03.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:03.142 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:07:03.192 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:07:03.192 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:07:03.266 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:07:03.267 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:07:03.267 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:07:03.268 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:07:03.268 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:07:03.268 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:07:03.269 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:03.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:03.269 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:07:03.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:03.270 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:07:03.270 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:03.270 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:03.270 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910022.283419_37.1", 2022-01-11T14:07:03.271 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:03.271 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:07:03.271 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:03.271 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:07:02.283419Z", 2022-01-11T14:07:03.272 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:03.272 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:03.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:03.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:03.273 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:07:03.273 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:03.274 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:03.274 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:07:03.274 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:03.274 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:03.275 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:03.275 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:07:03.275 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:03.275 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:03.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:03.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:03.276 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:07:03.276 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:03.276 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:03.277 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:07:03.277 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:03.277 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:03.277 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:03.278 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:07:03.278 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:03.278 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:03.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:03.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:03.279 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:07:03.279 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:03.279 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:03.279 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:07:03.280 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:03.280 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:03.280 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:03.280 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:07:03.281 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:03.281 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:03.281 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:03.281 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:07:03.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:03.282 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:07:03.282 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:07:03.282 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:07:03.282 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:03.283 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:07:03.284 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910022.347485_48.1 target=1_1641910022.283419_37.1 2022-01-11T14:07:03.691 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:07:03.693+0000 7fc1fb900700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:03.691 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:07:03.693+0000 7fc1fa0fd700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:03.691 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:07:03.693+0000 7fc1f28ee700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:03.692 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:07:03.695+0000 7fc1fc902700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:03.990 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:07:03.992+0000 7fc1f10eb700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:03.991 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:07:03.992+0000 7fc1f10eb700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:03.991 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:07:03.992+0000 7fc1f10eb700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:03.991 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:07:03.993+0000 7fc1f10eb700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:07:08.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:08.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:08.287 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:07:08.331 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:07:08.331 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:07:08.391 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:07:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:07:08.391 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:07:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:07:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:07:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:07:08.392 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:07:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:07:08.393 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910022.283419_37.1", 2022-01-11T14:07:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:08.394 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:07:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:07:02.283419Z", 2022-01-11T14:07:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:08.395 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:07:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:08.396 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:07:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:08.397 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:07:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:08.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:07:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:07:08.399 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:07:08.400 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:08.401 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:07:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:08.402 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:07:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:07:08.403 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:07:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:08.404 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:07:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:07:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:07:08.405 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:08.405 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:07:08.406 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910022.347485_48.1 target=1_1641910022.283419_37.1 2022-01-11T14:07:13.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:13.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:13.408 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:07:13.457 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:07:13.458 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:07:13.538 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:07:13.539 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:07:13.539 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:07:13.539 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:07:13.539 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:07:13.540 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:07:13.540 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:13.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:13.540 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:07:13.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:13.541 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:07:13.541 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:13.541 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:13.541 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910022.347485_48.1", 2022-01-11T14:07:13.542 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:13.542 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:07:13.542 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:13.543 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:07:02.347485Z", 2022-01-11T14:07:13.543 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:13.543 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:13.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:13.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:13.544 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:07:13.544 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:13.544 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:13.545 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:07:13.545 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:13.545 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:13.546 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:13.546 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:07:13.546 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:13.546 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:13.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:13.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:13.547 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:07:13.547 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:13.547 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:13.548 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:07:13.548 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:13.548 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:13.548 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:13.549 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:07:13.549 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:13.549 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:13.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:13.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:13.550 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:07:13.550 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:13.550 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:13.550 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:07:13.551 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:13.551 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:13.551 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:13.551 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:07:13.552 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:13.552 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:13.552 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:13.552 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:07:13.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:13.553 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:07:13.553 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:07:13.553 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:07:13.554 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:13.554 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:07:13.555 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:07:13.556 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:07:13.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:13.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:13.557 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:07:13.606 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:07:13.607 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:07:13.678 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:07:13.678 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:07:13.679 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:07:13.679 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:07:13.679 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:07:13.679 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:07:13.680 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:07:13.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:07:13.680 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:07:13.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:07:13.681 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:07:13.681 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:07:13.681 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:07:13.681 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910022.347485_48.1", 2022-01-11T14:07:13.681 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:07:13.682 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:07:13.682 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:07:13.682 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:07:02.347485Z", 2022-01-11T14:07:13.682 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:07:13.683 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:07:13.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:07:13.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:07:13.683 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:07:13.684 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:07:13.684 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:07:13.684 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:07:13.684 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:07:13.684 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:07:13.685 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:07:13.685 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:07:13.685 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:07:13.685 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:07:13.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:07:13.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:07:13.686 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:07:13.686 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:07:13.687 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:07:13.687 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:07:13.687 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:07:13.687 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:07:13.688 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:07:13.688 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:07:13.688 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:07:13.689 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:07:13.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:07:13.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:07:13.689 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:07:13.690 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:07:13.690 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:07:13.690 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:07:13.690 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:07:13.691 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:07:13.691 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:07:13.691 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:07:13.691 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:07:13.692 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:07:13.692 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:07:13.692 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:07:13.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:07:13.692 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:07:13.693 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:07:13.693 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:07:13.693 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:07:13.693 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:07:13.695 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:07:13.695 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:07:13.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:13.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:07:13.697 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:07:13.741 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:07:13.741 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:07:13.817 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:07:13.818 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:07:13.818 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:07:13.818 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:07:13.819 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:07:13.819 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:07:13.819 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:13.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:13.819 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:07:13.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:13.820 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:07:13.820 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:13.820 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:13.821 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910022.347485_48.1", 2022-01-11T14:07:13.821 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:13.821 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:07:13.821 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:13.821 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:07:02.347485Z", 2022-01-11T14:07:13.822 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:13.822 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:13.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:13.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:13.823 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:07:13.823 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:13.823 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:13.823 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:07:13.823 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:13.824 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:13.824 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:13.824 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:07:13.824 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:13.824 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:13.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:13.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:13.825 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:07:13.825 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:13.826 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:13.826 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:07:13.826 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:13.826 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:13.826 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:13.827 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:07:13.827 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:13.827 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:13.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:13.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:07:13.828 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:07:13.828 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:07:13.828 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:07:13.829 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:07:13.829 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:07:13.829 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:07:13.829 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:07:13.830 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:07:13.830 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:07:13.830 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:13.830 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:13.830 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:07:13.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:07:13.831 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:07:13.831 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:07:13.831 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:07:13.831 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:07:13.832 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:07:13.833 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:07:13.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:07:13.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:07:13.835 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:07:13.885 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:07:13.885 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:07:13.940 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:07:13.942+0000 7f8b52a71500 1 bucket sync caught up with source: 2022-01-11T14:07:13.940 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:13.941 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:13.941 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:07:13.942+0000 7f8b52a71500 0 bucket checkpoint complete 2022-01-11T14:07:13.952 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-10 zones={test-zone2, test-zone1} 2022-01-11T14:07:13.959 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:07:13.966 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-01-11T14:07:13.966 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-01-11T14:07:13.967 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-01-11T14:07:13.967 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:07:13.968 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:07:13.972 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-01-11T14:07:13.972 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-01-11T14:07:13.973 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-01-11T14:07:13.973 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:07:13.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:07:14.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:07:14.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:07:14.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:07:14.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:07:14.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:07:14.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:07:14.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:07:14.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:07:14.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:07:14.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:07:14.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:07:15.043 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-10 zones={test-zone2, test-zone1} 2022-01-11T14:07:15.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:07:15.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:07:15.045 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:07:15.102 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:07:15.102 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:07:15.162 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:07:15.164+0000 7f7a95d47500 1 bucket sync caught up with source: 2022-01-11T14:07:15.162 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:15.163 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:15.163 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:07:15.164+0000 7f7a95d47500 0 bucket checkpoint complete 2022-01-11T14:07:15.180 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-10 zones={test-zone3, test-zone1} 2022-01-11T14:07:15.187 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:07:15.191 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-01-11T14:07:15.192 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-01-11T14:07:15.193 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-01-11T14:07:15.193 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:07:15.194 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:07:15.197 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-01-11T14:07:15.198 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-01-11T14:07:15.198 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-01-11T14:07:15.199 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:07:15.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:07:15.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:07:15.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:07:15.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:07:15.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:07:15.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:07:15.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:07:15.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:07:15.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:07:15.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:07:16.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:07:16.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:07:16.214 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-10 zones={test-zone3, test-zone1} 2022-01-11T14:07:16.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:07:16.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:07:16.216 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:07:16.273 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:07:16.274 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:07:16.339 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:07:16.339+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:07:16.339 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:16.339 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:21.339 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:07:21.340+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:07:21.339 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:21.339 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:26.340 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:07:26.342+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:07:26.340 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:26.341 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:31.341 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:07:31.343+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:07:31.342 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:31.342 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:32.045 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:07:32.047+0000 7fc1e58d4700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2022-01-11T14:07:36.342 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:07:36.345+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:07:36.342 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:36.343 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:41.343 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:07:41.345+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:07:41.344 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:41.344 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:46.344 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:07:46.346+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:07:46.346 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:46.346 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:51.349 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:07:51.347+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:07:51.350 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:51.359 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:56.347 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:07:56.348+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:07:56.347 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:07:56.348 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:01.348 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:08:01.350+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:08:01.348 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:01.349 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:06.349 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:08:06.352+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:08:06.350 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:06.350 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:11.350 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:08:11.353+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:08:11.351 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:11.351 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:16.352 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:08:16.354+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:08:16.352 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:16.353 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:21.353 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:08:21.354+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:08:21.353 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:21.353 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:25.350 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:08:25.353+0000 7fc1e3ebe700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2022-01-11T14:08:26.354 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:08:26.356+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:08:26.355 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:26.355 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:31.356 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:08:31.357+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:08:31.356 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:31.357 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:36.357 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:08:36.358+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:08:36.357 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:36.357 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:41.358 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:08:41.360+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:08:41.358 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:41.359 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:46.359 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:08:46.362+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:08:46.359 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:46.360 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:51.360 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:08:51.363+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:08:51.361 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:51.361 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:56.362 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:08:56.364+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:08:56.362 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:08:56.362 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:01.363 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:09:01.365+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:09:01.363 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:01.364 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:06.364 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:09:06.366+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:09:06.365 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:06.365 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:11.365 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:09:11.367+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:09:11.366 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:11.366 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:16.367 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:09:16.368+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:09:16.367 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:16.368 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:21.368 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:09:21.369+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:09:21.368 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:21.368 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:26.369 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:09:26.372+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:09:26.369 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:26.369 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:31.370 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:09:31.373+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:09:31.371 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:31.371 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:36.371 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:09:36.374+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:09:36.372 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:36.372 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:41.373 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:09:41.375+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:09:41.373 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:41.373 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:46.374 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:09:46.375+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:09:46.375 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:46.375 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:51.375 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:09:51.377+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:09:51.375 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:51.376 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:56.376 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:09:56.378+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:09:56.377 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:09:56.377 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:10:01.377 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:01.379+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:10:01.378 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:10:01.378 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:10:06.379 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:06.381+0000 7f9cd9d1c500 1 waiting for incremental sync to catch up: 2022-01-11T14:10:06.379 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , , , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:10:06.379 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:10:10.558 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:10:10.561+0000 7f3d897c4700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2022-01-11T14:10:11.380 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:11.383+0000 7f9cd9d1c500 1 bucket sync caught up with source: 2022-01-11T14:10:11.381 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:10:11.381 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.9.5, , 00000000001.7.5, , , , 00000000001.7.5, 00000000001.4.5, , ] 2022-01-11T14:10:11.381 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:11.383+0000 7f9cd9d1c500 0 bucket checkpoint complete 2022-01-11T14:10:11.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:11.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:11.406 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:11.464 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:11.464 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:11.534 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:11.537+0000 7ff9208d7500 1 bucket sync caught up with source: 2022-01-11T14:10:11.535 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000001.25.5, , 00000000001.14.5, , , , 00000000001.35.5, 00000000001.32.5, , ] 2022-01-11T14:10:11.535 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000001.25.5, , 00000000001.14.5, , , , 00000000001.35.5, 00000000001.32.5, , ] 2022-01-11T14:10:11.535 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:11.537+0000 7ff9208d7500 0 bucket checkpoint complete 2022-01-11T14:10:11.552 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-11 zones={test-zone1, test-zone2} 2022-01-11T14:10:11.565 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:11.570 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-01-11T14:10:11.571 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-01-11T14:10:11.572 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-01-11T14:10:11.573 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:11.573 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:11.580 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-01-11T14:10:11.583 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-01-11T14:10:11.584 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-01-11T14:10:11.585 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:11.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:10:11.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:10:11.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:10:11.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:10:11.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:10:12.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:10:12.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:10:12.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:10:12.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:10:12.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:12.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:12.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:12.656 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-11 zones={test-zone1, test-zone2} 2022-01-11T14:10:12.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:12.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:12.658 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:12.710 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:12.710 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:12.760 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:12.763+0000 7fa68b20b500 1 bucket sync caught up with source: 2022-01-11T14:10:12.760 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000001.25.5, , 00000000001.14.5, , , , 00000000001.35.5, 00000000001.32.5, , ] 2022-01-11T14:10:12.760 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000001.25.5, , 00000000001.14.5, , , , 00000000001.35.5, 00000000001.32.5, , ] 2022-01-11T14:10:12.761 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:12.763+0000 7fa68b20b500 0 bucket checkpoint complete 2022-01-11T14:10:12.776 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-11 zones={test-zone3, test-zone2} 2022-01-11T14:10:12.785 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:12.789 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-01-11T14:10:12.789 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-01-11T14:10:12.790 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-01-11T14:10:12.790 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:12.791 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:12.796 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-01-11T14:10:12.796 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-01-11T14:10:12.797 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-01-11T14:10:12.798 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:12.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:10:12.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:10:12.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:10:13.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:10:13.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:10:13.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:10:13.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:10:13.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:10:13.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:10:13.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:13.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:13.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:13.996 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-11 zones={test-zone3, test-zone2} 2022-01-11T14:10:13.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:13.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:13.998 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:14.064 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:14.065 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:14.126 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:14.129+0000 7fd41a723500 1 bucket sync caught up with source: 2022-01-11T14:10:14.126 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000001.55.5, , 00000000001.58.5, , , , 00000000001.22.5, 00000000001.73.5, , ] 2022-01-11T14:10:14.126 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000001.55.5, , 00000000001.58.5, , , , 00000000001.22.5, 00000000001.73.5, , ] 2022-01-11T14:10:14.127 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:14.129+0000 7fd41a723500 0 bucket checkpoint complete 2022-01-11T14:10:14.142 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-12 zones={test-zone1, test-zone3} 2022-01-11T14:10:14.157 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:14.166 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-01-11T14:10:14.167 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-01-11T14:10:14.167 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-01-11T14:10:14.168 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:14.168 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:14.173 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-01-11T14:10:14.174 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-01-11T14:10:14.174 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-01-11T14:10:14.175 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:14.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:10:14.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:10:14.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:10:14.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:10:14.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:10:14.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:10:14.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:10:14.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:10:14.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:10:15.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:15.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:15.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:15.274 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-12 zones={test-zone1, test-zone3} 2022-01-11T14:10:15.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:15.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:15.276 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:15.338 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:15.339 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:15.413 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:15.416+0000 7f92fa0fe500 1 bucket sync caught up with source: 2022-01-11T14:10:15.414 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.55.5, , 00000000001.58.5, , , , 00000000001.22.5, 00000000001.73.5, , ] 2022-01-11T14:10:15.414 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.55.5, , 00000000001.58.5, , , , 00000000001.22.5, 00000000001.73.5, , ] 2022-01-11T14:10:15.414 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:15.416+0000 7f92fa0fe500 0 bucket checkpoint complete 2022-01-11T14:10:15.427 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-12 zones={test-zone2, test-zone3} 2022-01-11T14:10:15.434 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:15.438 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-01-11T14:10:15.438 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-01-11T14:10:15.439 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-01-11T14:10:15.439 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:15.440 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:15.445 DEBUG:tasks.rgw_multi.zone_rados:o=& 2022-01-11T14:10:15.445 DEBUG:tasks.rgw_multi.zone_rados:o=: 2022-01-11T14:10:15.446 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2022-01-11T14:10:15.446 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:15.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:10:15.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:10:15.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2022-01-11T14:10:15.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:10:15.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:10:15.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2022-01-11T14:10:15.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:10:16.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:10:16.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2022-01-11T14:10:16.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:16.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:16.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:16.495 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-12 zones={test-zone2, test-zone3} 2022-01-11T14:10:16.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2022-01-11T14:10:16.497 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-13 2022-01-11T14:10:16.511 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-14 2022-01-11T14:10:16.541 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-15 2022-01-11T14:10:16.696 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2022-01-11T14:10:16.699+0000 7f98cb045700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:10:16.697 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:10:16.699+0000 7f3d94bf4700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:10:16.697 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:10:16.699+0000 7fc1f70f7700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:10:16.698 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:10:16.701+0000 7f3d97bfa700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:10:16.712 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2022-01-11T14:10:16.715+0000 7f98cb846700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:10:16.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:16.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:16.729 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:10:16.772 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:16.772 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:16.829 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:10:16.843 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:10:16.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:16.843 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910216.567518_66.1", 2022-01-11T14:10:16.844 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:10:16.567518Z" 2022-01-11T14:10:16.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:16.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:16.844 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:10:16.845 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:10:16.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:16.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:16.845 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:10:16.846 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:10:16.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:16.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:16.846 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:10:16.846 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:10:16.847 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:10:16.847 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:10:16.847 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:10:16.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:16.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:16.849 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:10:16.894 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:16.894 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:16.952 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:10:16.952 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:10:16.952 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:10:16.952 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:10:16.953 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:10:16.953 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:10:16.953 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:16.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:16.954 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:10:16.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:16.954 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:10:16.954 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:16.955 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:16.955 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910022.347485_48.1", 2022-01-11T14:10:16.955 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:16.955 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:10:16.956 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:16.956 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:07:02.347485Z", 2022-01-11T14:10:16.956 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:16.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:10:16.957 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:16.958 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:16.958 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:10:16.958 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:16.958 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:16.958 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:16.959 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:10:16.959 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:16.959 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:16.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:16.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:16.960 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:10:16.960 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:16.961 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:16.961 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:10:16.961 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:16.961 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:16.961 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:16.962 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:10:16.962 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:16.962 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:16.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:16.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:16.963 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:10:16.963 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:16.963 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:16.963 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:10:16.964 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:16.964 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:16.964 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:16.964 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:10:16.965 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:16.965 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:16.965 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:16.965 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:10:16.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:16.966 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:10:16.966 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:10:16.966 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:10:16.967 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:16.967 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:10:16.968 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910216.567518_66.1 target=1_1641910022.347485_48.1 2022-01-11T14:10:21.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:21.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:21.971 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:10:22.022 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:22.023 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:22.085 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:10:22.085 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:10:22.085 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:10:22.085 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:10:22.086 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:10:22.086 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:10:22.086 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:22.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:22.087 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:10:22.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:22.087 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:10:22.087 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:22.088 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:22.088 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910216.567518_66.1", 2022-01-11T14:10:22.088 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:22.088 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:10:22.088 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:22.089 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:10:16.567518Z", 2022-01-11T14:10:22.089 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:22.089 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:22.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:22.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:22.090 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:10:22.090 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:22.090 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:22.091 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:10:22.091 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:22.091 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:22.091 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:22.091 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:10:22.092 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:22.092 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:22.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:22.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:22.095 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:10:22.096 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:22.096 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:22.096 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:10:22.097 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:22.097 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:22.098 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:22.099 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:10:22.099 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:22.101 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:22.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:22.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:22.102 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:10:22.102 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:22.102 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:22.103 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:10:22.103 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:22.103 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:22.103 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:22.103 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:10:22.104 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:22.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:22.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:22.104 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:10:22.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:22.105 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:10:22.105 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:10:22.105 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:10:22.105 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:22.106 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:10:22.108 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:10:22.108 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:10:22.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:22.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:22.111 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:10:22.162 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:22.163 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:22.232 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:10:22.232 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:10:22.233 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:10:22.233 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:10:22.233 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:10:22.233 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:10:22.235 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:10:22.235 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:22.235 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:10:22.236 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:22.236 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:10:22.236 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:10:22.237 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:10:22.237 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910216.567518_66.1", 2022-01-11T14:10:22.237 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:10:22.237 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:10:22.238 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:10:22.238 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:10:16.567518Z", 2022-01-11T14:10:22.238 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:10:22.238 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:10:22.239 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:22.239 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:22.239 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:10:22.239 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:10:22.239 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:10:22.240 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:10:22.240 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:10:22.240 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:10:22.240 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:10:22.241 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:10:22.241 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:10:22.242 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:10:22.242 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:22.243 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:22.243 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:10:22.243 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:10:22.243 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:10:22.244 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:10:22.244 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:10:22.244 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:10:22.244 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:10:22.244 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:10:22.245 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:10:22.245 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:10:22.245 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:22.245 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:22.246 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:10:22.246 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:10:22.247 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:10:22.247 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:10:22.247 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:10:22.248 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:10:22.248 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:10:22.248 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:10:22.248 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:10:22.248 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:10:22.249 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:10:22.249 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:10:22.249 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:22.249 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:10:22.250 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:10:22.250 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:10:22.250 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:10:22.250 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:10:22.252 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:10:22.252 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:10:22.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:22.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:22.254 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:10:22.298 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:22.299 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:22.365 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:10:22.366 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:10:22.366 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:10:22.366 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:10:22.367 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:10:22.367 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:10:22.367 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:22.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:22.368 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:10:22.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:22.368 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:10:22.368 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:22.368 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:22.369 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910216.567518_66.1", 2022-01-11T14:10:22.369 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:22.369 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:10:22.369 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:22.370 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:10:16.567518Z", 2022-01-11T14:10:22.370 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:22.371 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:22.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:22.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:22.371 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:10:22.372 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:22.372 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:22.372 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:10:22.372 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:22.373 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:22.373 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:22.373 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:10:22.373 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:22.373 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:22.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:22.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:22.374 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:10:22.374 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:22.375 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:22.375 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:10:22.375 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:22.375 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:22.376 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:22.376 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:10:22.376 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:22.376 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:22.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:22.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:22.377 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:10:22.377 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:22.378 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:22.378 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:10:22.378 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:22.378 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:22.378 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:22.379 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:10:22.379 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:22.380 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:22.380 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:22.380 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:10:22.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:22.381 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:10:22.381 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:10:22.381 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:10:22.381 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:22.382 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:10:22.383 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:10:22.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:22.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:22.385 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:22.473 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:22.473 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:22.531 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:22.534+0000 7f22b5a97500 1 bucket sync caught up with source: 2022-01-11T14:10:22.533 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2022-01-11T14:10:22.533 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2022-01-11T14:10:22.533 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:22.534+0000 7f22b5a97500 0 bucket checkpoint complete 2022-01-11T14:10:22.548 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-13 zones={test-zone2, test-zone1} 2022-01-11T14:10:22.555 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:22.599 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:22.600 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:22.603 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:22.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:22.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:22.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:22.922 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-13 zones={test-zone2, test-zone1} 2022-01-11T14:10:22.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:22.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:22.924 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:22.978 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:22.978 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:23.041 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:23.044+0000 7fa090a92500 1 bucket sync caught up with source: 2022-01-11T14:10:23.041 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2022-01-11T14:10:23.041 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2022-01-11T14:10:23.042 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:23.044+0000 7fa090a92500 0 bucket checkpoint complete 2022-01-11T14:10:23.059 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-13 zones={test-zone3, test-zone1} 2022-01-11T14:10:23.065 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:23.111 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:23.112 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:23.118 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:23.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:23.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:23.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:23.427 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-13 zones={test-zone3, test-zone1} 2022-01-11T14:10:23.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:23.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:23.429 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:23.482 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:23.483 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:23.538 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:23.538+0000 7f449702f500 1 bucket sync caught up with source: 2022-01-11T14:10:23.539 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2022-01-11T14:10:23.539 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2022-01-11T14:10:23.539 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:23.538+0000 7f449702f500 0 bucket checkpoint complete 2022-01-11T14:10:23.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:23.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:23.550 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:23.600 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:23.601 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:23.647 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:23.650+0000 7f0394866500 1 bucket sync caught up with source: 2022-01-11T14:10:23.647 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000001.25.5, , , , , , , , , ] 2022-01-11T14:10:23.647 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000001.25.5, , , , , , , , , ] 2022-01-11T14:10:23.647 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:23.650+0000 7f0394866500 0 bucket checkpoint complete 2022-01-11T14:10:23.659 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-14 zones={test-zone1, test-zone2} 2022-01-11T14:10:23.665 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:23.715 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:23.716 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:23.719 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:23.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:23.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:23.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:24.035 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-14 zones={test-zone1, test-zone2} 2022-01-11T14:10:24.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:24.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:24.037 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:24.088 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:24.110 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:24.138 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:24.141+0000 7f448443b500 1 bucket sync caught up with source: 2022-01-11T14:10:24.139 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000001.25.5, , , , , , , , , ] 2022-01-11T14:10:24.139 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000001.25.5, , , , , , , , , ] 2022-01-11T14:10:24.139 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:24.141+0000 7f448443b500 0 bucket checkpoint complete 2022-01-11T14:10:24.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-14 zones={test-zone3, test-zone2} 2022-01-11T14:10:24.169 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:24.220 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:24.221 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:24.225 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:24.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:24.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:24.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:24.538 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-14 zones={test-zone3, test-zone2} 2022-01-11T14:10:24.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:24.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:24.540 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:24.591 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:24.592 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:24.653 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:24.656+0000 7f1016bb7500 1 bucket sync caught up with source: 2022-01-11T14:10:24.653 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000001.154.5, , , , , , , , , ] 2022-01-11T14:10:24.653 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000001.154.5, , , , , , , , , ] 2022-01-11T14:10:24.654 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:24.656+0000 7f1016bb7500 0 bucket checkpoint complete 2022-01-11T14:10:24.678 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-15 zones={test-zone1, test-zone3} 2022-01-11T14:10:24.685 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:24.731 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:24.732 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:24.737 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:24.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:24.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:24.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:25.047 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-15 zones={test-zone1, test-zone3} 2022-01-11T14:10:25.048 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:25.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:25.049 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:25.107 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:25.107 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:25.166 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:25.168+0000 7f6c9bce3500 1 bucket sync caught up with source: 2022-01-11T14:10:25.167 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.154.5, , , , , , , , , ] 2022-01-11T14:10:25.167 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.154.5, , , , , , , , , ] 2022-01-11T14:10:25.167 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:25.168+0000 7f6c9bce3500 0 bucket checkpoint complete 2022-01-11T14:10:25.184 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-15 zones={test-zone2, test-zone3} 2022-01-11T14:10:25.191 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:25.243 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:25.244 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:25.248 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2022-01-11T14:10:25.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:25.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:25.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2022-01-11T14:10:25.562 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-15 zones={test-zone2, test-zone3} 2022-01-11T14:10:25.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:25.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:25.574 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:25.626 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:25.626 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:26.689 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:26.691+0000 7f333a6b2500 1 bucket sync caught up with source: 2022-01-11T14:10:26.689 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2022-01-11T14:10:26.690 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2022-01-11T14:10:26.690 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:26.691+0000 7f333a6b2500 0 bucket checkpoint complete 2022-01-11T14:10:26.705 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-13 zones={test-zone2, test-zone1} 2022-01-11T14:10:26.715 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:26.767 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:26.827 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-13 zones={test-zone2, test-zone1} 2022-01-11T14:10:26.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:26.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:26.829 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:26.884 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:26.884 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:26.956 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:26.959+0000 7f2bace6e500 1 bucket sync caught up with source: 2022-01-11T14:10:26.957 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2022-01-11T14:10:26.957 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2022-01-11T14:10:26.957 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:26.959+0000 7f2bace6e500 0 bucket checkpoint complete 2022-01-11T14:10:26.979 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-13 zones={test-zone3, test-zone1} 2022-01-11T14:10:26.985 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:27.035 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:27.090 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-13 zones={test-zone3, test-zone1} 2022-01-11T14:10:27.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:27.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:27.092 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:27.148 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:27.149 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:27.197 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:27.199+0000 7f5393c8b500 1 bucket sync caught up with source: 2022-01-11T14:10:27.198 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2022-01-11T14:10:27.198 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2022-01-11T14:10:27.198 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:27.199+0000 7f5393c8b500 0 bucket checkpoint complete 2022-01-11T14:10:27.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:27.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:27.231 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:27.282 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:27.282 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:27.341 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:27.344+0000 7f6c65c38500 1 waiting for incremental sync to catch up: 2022-01-11T14:10:27.341 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000001.25.5, , , , , , , , , ] 2022-01-11T14:10:27.342 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000002.57.5, , , , , , , , , ] 2022-01-11T14:10:32.342 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:32.345+0000 7f6c65c38500 1 bucket sync caught up with source: 2022-01-11T14:10:32.343 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000002.57.5, , , , , , , , , ] 2022-01-11T14:10:32.343 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000002.57.5, , , , , , , , , ] 2022-01-11T14:10:32.343 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:32.345+0000 7f6c65c38500 0 bucket checkpoint complete 2022-01-11T14:10:32.359 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-14 zones={test-zone1, test-zone2} 2022-01-11T14:10:32.371 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:32.419 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:32.474 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-14 zones={test-zone1, test-zone2} 2022-01-11T14:10:32.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:32.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:32.476 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:32.532 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:32.533 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:32.585 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:32.588+0000 7f346c976500 1 bucket sync caught up with source: 2022-01-11T14:10:32.585 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000002.57.5, , , , , , , , , ] 2022-01-11T14:10:32.585 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000002.57.5, , , , , , , , , ] 2022-01-11T14:10:32.586 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:32.588+0000 7f346c976500 0 bucket checkpoint complete 2022-01-11T14:10:32.605 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-14 zones={test-zone3, test-zone2} 2022-01-11T14:10:32.612 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:32.659 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:32.714 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-14 zones={test-zone3, test-zone2} 2022-01-11T14:10:32.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:32.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:32.727 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:32.777 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:32.778 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:32.842 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:32.845+0000 7fd57dc5f500 1 waiting for incremental sync to catch up: 2022-01-11T14:10:32.843 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000001.154.5, , , , , , , , , ] 2022-01-11T14:10:32.843 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000002.196.5, , , , , , , , , ] 2022-01-11T14:10:37.843 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:37.846+0000 7fd57dc5f500 1 bucket sync caught up with source: 2022-01-11T14:10:37.843 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000002.196.5, , , , , , , , , ] 2022-01-11T14:10:37.843 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000002.196.5, , , , , , , , , ] 2022-01-11T14:10:37.844 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:37.846+0000 7fd57dc5f500 0 bucket checkpoint complete 2022-01-11T14:10:37.860 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-15 zones={test-zone1, test-zone3} 2022-01-11T14:10:37.867 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:37.915 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:37.967 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-15 zones={test-zone1, test-zone3} 2022-01-11T14:10:37.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:37.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:37.969 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:38.028 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:38.028 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:38.080 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:38.082+0000 7f8b0c894500 1 bucket sync caught up with source: 2022-01-11T14:10:38.081 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.196.5, , , , , , , , , ] 2022-01-11T14:10:38.081 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.196.5, , , , , , , , , ] 2022-01-11T14:10:38.081 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:38.082+0000 7f8b0c894500 0 bucket checkpoint complete 2022-01-11T14:10:38.095 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-15 zones={test-zone2, test-zone3} 2022-01-11T14:10:38.102 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:10:38.147 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:10:38.199 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-15 zones={test-zone2, test-zone3} 2022-01-11T14:10:38.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2022-01-11T14:10:38.200 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-16 2022-01-11T14:10:38.217 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-17 2022-01-11T14:10:38.243 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-18 2022-01-11T14:10:38.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:38.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:38.319 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:10:38.363 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:38.363 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:38.414 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:10:38.433 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:10:38.433 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:38.434 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910238.318780_84.1", 2022-01-11T14:10:38.434 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:10:38.318780Z" 2022-01-11T14:10:38.434 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:38.434 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:38.434 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:10:38.435 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:10:38.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:38.435 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:38.435 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:10:38.436 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:10:38.436 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:38.436 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:38.436 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:10:38.437 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:10:38.437 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:10:38.437 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:10:38.438 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:10:38.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:38.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:38.440 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:10:38.483 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:38.484 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:38.556 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:10:38.557 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:10:38.557 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:10:38.557 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:10:38.557 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:10:38.558 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:10:38.558 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:38.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:38.558 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:10:38.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:38.559 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:10:38.559 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:38.560 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:38.560 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910238.210468_67.1", 2022-01-11T14:10:38.560 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:38.560 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:10:38.561 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:38.561 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:10:38.210468Z", 2022-01-11T14:10:38.561 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:38.561 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:38.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:38.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:38.562 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:10:38.562 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:38.562 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:38.563 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:10:38.563 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:38.563 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:38.563 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:38.564 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:10:38.564 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:38.564 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:38.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:38.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:38.565 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:10:38.565 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:38.565 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:38.565 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:10:38.566 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:38.566 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:38.566 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:38.566 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:10:38.566 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:38.567 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:38.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:38.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:38.567 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:10:38.568 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:38.568 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:38.568 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:10:38.568 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:38.568 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:38.569 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:38.569 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:10:38.569 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:38.569 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:38.570 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:38.570 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:10:38.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:38.571 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:10:38.571 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:10:38.571 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:10:38.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:38.572 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:10:38.573 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910238.318780_84.1 target=1_1641910238.210468_67.1 2022-01-11T14:10:43.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:43.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:43.575 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:10:43.626 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:43.627 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:43.705 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:10:43.705 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:10:43.705 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:10:43.706 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:10:43.706 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:10:43.706 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:10:43.707 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:43.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:43.707 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:10:43.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:43.707 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:10:43.708 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:43.708 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:43.708 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910238.318780_84.1", 2022-01-11T14:10:43.708 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:43.708 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:10:43.709 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:43.709 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:10:38.318780Z", 2022-01-11T14:10:43.709 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:43.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:43.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:43.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:43.710 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:10:43.710 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:43.711 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:43.711 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:10:43.711 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:43.711 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:43.711 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:43.712 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:10:43.712 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:43.712 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:43.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:43.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:43.713 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:10:43.714 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:43.714 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:43.714 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:10:43.714 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:43.714 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:43.715 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:43.715 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:10:43.715 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:43.716 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:43.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:43.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:43.716 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:10:43.717 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:43.717 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:43.717 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:10:43.717 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:43.718 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:43.718 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:43.718 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:10:43.718 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:43.718 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:43.719 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:43.719 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:10:43.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:43.719 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:10:43.720 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:10:43.720 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:10:43.720 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:43.720 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:10:43.722 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:10:43.722 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:10:43.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:43.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:43.724 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:10:43.774 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:43.775 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:43.843 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:10:43.844 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:10:43.844 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:10:43.844 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:10:43.844 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:10:43.845 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:10:43.845 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:10:43.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:43.845 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:10:43.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:43.846 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:10:43.846 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:10:43.846 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:10:43.846 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910238.318780_84.1", 2022-01-11T14:10:43.847 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:10:43.847 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:10:43.847 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:10:43.847 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:10:38.318780Z", 2022-01-11T14:10:43.848 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:10:43.848 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:10:43.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:43.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:43.848 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:10:43.849 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:10:43.849 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:10:43.849 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:10:43.849 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:10:43.850 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:10:43.850 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:10:43.850 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:10:43.850 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:10:43.851 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:10:43.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:43.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:43.851 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:10:43.852 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:10:43.852 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:10:43.852 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:10:43.852 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:10:43.853 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:10:43.853 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:10:43.853 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:10:43.853 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:10:43.853 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:10:43.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:43.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:10:43.854 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:10:43.854 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:10:43.855 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:10:43.855 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:10:43.855 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:10:43.855 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:10:43.856 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:10:43.856 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:10:43.856 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:10:43.856 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:10:43.856 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:10:43.857 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:10:43.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:10:43.857 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:10:43.857 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:10:43.858 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:10:43.858 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:10:43.858 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:10:43.859 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:10:43.860 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:10:43.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:43.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:10:43.861 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:10:43.906 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:43.906 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:43.968 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:10:43.968 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:10:43.969 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:10:43.969 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:10:43.969 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:10:43.969 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:10:43.970 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:43.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:43.970 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:10:43.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:43.971 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:10:43.971 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:43.971 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:43.971 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910238.318780_84.1", 2022-01-11T14:10:43.972 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:43.972 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:10:43.972 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:43.972 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:10:38.318780Z", 2022-01-11T14:10:43.972 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:43.973 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:43.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:43.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:43.973 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:10:43.974 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:43.974 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:43.974 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:10:43.974 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:43.975 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:43.975 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:43.975 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:10:43.975 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:43.975 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:43.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:43.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:43.976 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:10:43.976 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:43.977 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:43.977 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:10:43.977 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:43.977 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:43.978 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:43.978 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:10:43.978 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:43.978 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:43.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:43.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:10:43.979 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:10:43.979 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:10:43.979 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:10:43.980 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:10:43.980 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:10:43.980 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:10:43.980 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:10:43.981 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:10:43.981 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:10:43.981 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:43.981 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:43.981 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:10:43.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:10:43.982 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:10:43.982 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:10:43.982 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:10:43.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:10:43.983 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:10:43.984 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:10:44.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:44.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:44.036 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:44.086 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:44.135 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:44.137+0000 7f6f07606500 1 waiting for incremental sync to catch up: 2022-01-11T14:10:44.136 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , , , ] 2022-01-11T14:10:44.136 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000002.11.10, , , , , ] 2022-01-11T14:10:49.137 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:49.139+0000 7f6f07606500 1 bucket sync caught up with source: 2022-01-11T14:10:49.137 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.17.9, , , , , ] 2022-01-11T14:10:49.137 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000002.11.10, , , , , ] 2022-01-11T14:10:49.137 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:49.139+0000 7f6f07606500 0 bucket checkpoint complete 2022-01-11T14:10:49.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:49.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:49.159 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:49.214 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:49.215 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:49.282 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:49.284+0000 7f28acd19500 1 bucket sync caught up with source: 2022-01-11T14:10:49.282 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , 00000000005.17.9, , , , , ] 2022-01-11T14:10:49.282 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , 00000000005.17.9, , , , , ] 2022-01-11T14:10:49.283 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:49.284+0000 7f28acd19500 0 bucket checkpoint complete 2022-01-11T14:10:49.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:49.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:49.306 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:49.362 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:49.363 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:49.415 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:49.418+0000 7fcd4199a500 1 bucket sync caught up with source: 2022-01-11T14:10:49.415 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.17.9, , , , , ] 2022-01-11T14:10:49.416 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000005.17.9, , , , , ] 2022-01-11T14:10:49.416 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:10:49.418+0000 7fcd4199a500 0 bucket checkpoint complete 2022-01-11T14:10:49.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:49.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:49.480 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:49.532 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:49.532 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:49.594 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:49.596+0000 7fede2b2b500 1 waiting for incremental sync to catch up: 2022-01-11T14:10:49.594 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , , , ] 2022-01-11T14:10:49.595 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , 00000000002.134.10, , , , , ] 2022-01-11T14:10:54.595 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:54.597+0000 7fede2b2b500 1 bucket sync caught up with source: 2022-01-11T14:10:54.596 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , 00000000005.140.9, , , , , ] 2022-01-11T14:10:54.596 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , 00000000002.134.10, , , , , ] 2022-01-11T14:10:54.596 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:54.597+0000 7fede2b2b500 0 bucket checkpoint complete 2022-01-11T14:10:54.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:54.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:54.620 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:54.673 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:54.674 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:54.741 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:54.743+0000 7f5d3ad00500 1 bucket sync caught up with source: 2022-01-11T14:10:54.742 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , 00000000005.140.9, , , , , ] 2022-01-11T14:10:54.742 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , 00000000005.140.9, , , , , ] 2022-01-11T14:10:54.742 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:54.743+0000 7f5d3ad00500 0 bucket checkpoint complete 2022-01-11T14:10:54.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:54.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:54.812 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:54.863 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:54.864 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:10:54.910 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:54.912+0000 7f0d20189500 1 waiting for incremental sync to catch up: 2022-01-11T14:10:54.911 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , , , ] 2022-01-11T14:10:54.911 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , 00000000002.43.10, , , , , ] 2022-01-11T14:10:59.911 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:59.913+0000 7f0d20189500 1 bucket sync caught up with source: 2022-01-11T14:10:59.912 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , 00000000005.49.9, , , , , ] 2022-01-11T14:10:59.912 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , 00000000002.43.10, , , , , ] 2022-01-11T14:10:59.912 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:10:59.913+0000 7f0d20189500 0 bucket checkpoint complete 2022-01-11T14:10:59.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:59.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:10:59.936 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:10:59.992 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:10:59.992 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:00.048 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:00.050+0000 7fb063418500 1 bucket sync caught up with source: 2022-01-11T14:11:00.048 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.49.9, , , , , ] 2022-01-11T14:11:00.048 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000005.49.9, , , , , ] 2022-01-11T14:11:00.048 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:00.050+0000 7fb063418500 0 bucket checkpoint complete 2022-01-11T14:11:01.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:01.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:01.678 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:01.731 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:01.731 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:01.803 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:01.805+0000 7f0fbc8fa500 1 bucket sync caught up with source: 2022-01-11T14:11:01.803 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000010.27.5, , 00000000003.12.9, , 00000000008.23.9, , , , , ] 2022-01-11T14:11:01.803 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000010.27.5, , 00000000003.12.9, , 00000000008.23.9, , , , , ] 2022-01-11T14:11:01.804 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:01.805+0000 7f0fbc8fa500 0 bucket checkpoint complete 2022-01-11T14:11:01.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:01.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:01.823 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:01.875 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:01.875 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:01.941+0000 7fe5cacd5500 1 bucket sync caught up with source: 2022-01-11T14:11:01.939 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000010.27.5, , 00000000003.12.9, , 00000000008.23.9, , , , , ] 2022-01-11T14:11:01.939 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000010.27.5, , 00000000003.12.9, , 00000000008.23.9, , , , , ] 2022-01-11T14:11:01.940 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:01.941+0000 7fe5cacd5500 0 bucket checkpoint complete 2022-01-11T14:11:01.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:01.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:01.963 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:02.016 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:02.016 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:02.076 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:02.079+0000 7f0f3d4d6500 1 bucket sync caught up with source: 2022-01-11T14:11:02.076 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000010.27.5, , 00000000003.12.9, , 00000000008.23.9, , , , , ] 2022-01-11T14:11:02.076 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000010.27.5, , 00000000003.12.9, , 00000000008.23.9, , , , , ] 2022-01-11T14:11:02.077 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:02.079+0000 7f0f3d4d6500 0 bucket checkpoint complete 2022-01-11T14:11:02.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:02.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:02.089 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:02.139 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:02.139 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:02.201 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:02.203+0000 7fc8565ad500 1 bucket sync caught up with source: 2022-01-11T14:11:02.201 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000006.213.5, , 00000000010.280.5, , 00000000007.206.9, , , , , ] 2022-01-11T14:11:02.202 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000006.213.5, , 00000000010.280.5, , 00000000007.206.9, , , , , ] 2022-01-11T14:11:02.202 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:02.203+0000 7fc8565ad500 0 bucket checkpoint complete 2022-01-11T14:11:02.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:02.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:02.223 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:02.275 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:02.320 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:02.322+0000 7f25f7a52500 1 bucket sync caught up with source: 2022-01-11T14:11:02.320 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000006.213.5, , 00000000010.280.5, , 00000000007.206.9, , , , , ] 2022-01-11T14:11:02.321 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000006.213.5, , 00000000010.280.5, , 00000000007.206.9, , , , , ] 2022-01-11T14:11:02.321 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:02.322+0000 7f25f7a52500 0 bucket checkpoint complete 2022-01-11T14:11:02.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:02.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:02.334 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:02.382 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:02.383 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:02.441 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:02.443+0000 7f38482f1500 1 bucket sync caught up with source: 2022-01-11T14:11:02.441 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000007.236.5, , 00000000003.300.9, , 00000000014.249.5, , , , , ] 2022-01-11T14:11:02.441 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000007.236.5, , 00000000003.300.9, , 00000000014.249.5, , , , , ] 2022-01-11T14:11:02.441 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:02.443+0000 7f38482f1500 0 bucket checkpoint complete 2022-01-11T14:11:02.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:02.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:02.459 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:02.509 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:02.509 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:02.554 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:02.557+0000 7fb1a10b2500 1 bucket sync caught up with source: 2022-01-11T14:11:02.554 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000007.236.5, , 00000000003.300.9, , 00000000014.249.5, , , , , ] 2022-01-11T14:11:02.554 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000007.236.5, , 00000000003.300.9, , 00000000014.249.5, , , , , ] 2022-01-11T14:11:02.554 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:02.557+0000 7fb1a10b2500 0 bucket checkpoint complete 2022-01-11T14:11:02.566 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-16 zones={test-zone2, test-zone1} 2022-01-11T14:11:02.573 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:02.577 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:02.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:02.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:02.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:02.579 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:02.583 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:02.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:02.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:02.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:02.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:02.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:02.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:02.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:02.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:02.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:02.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:03.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:03.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:03.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:03.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:03.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:03.491 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-16 zones={test-zone2, test-zone1} 2022-01-11T14:11:03.491 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-16 zones={test-zone3, test-zone1} 2022-01-11T14:11:03.498 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:03.502 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:03.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:03.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:03.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:03.504 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:03.508 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:03.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:03.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:03.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:03.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:03.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:03.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:03.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:03.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:03.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:03.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:03.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:04.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:04.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:04.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:04.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:04.451 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-16 zones={test-zone3, test-zone1} 2022-01-11T14:11:04.452 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-16 zones={test-zone3, test-zone2} 2022-01-11T14:11:04.462 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:04.466 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:04.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:04.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:04.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:04.468 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:04.472 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:04.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:04.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:04.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:04.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:04.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:04.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:04.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:04.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:04.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:04.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:04.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:05.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:05.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:05.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:05.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:05.379 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-16 zones={test-zone3, test-zone2} 2022-01-11T14:11:05.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:05.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:05.381 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:05.437 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:05.437 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:05.490 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:05.492+0000 7f41138a4500 1 bucket sync caught up with source: 2022-01-11T14:11:05.490 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000010.23.5, , 00000000003.14.9, , 00000000007.19.9, , , , , ] 2022-01-11T14:11:05.491 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000010.23.5, , 00000000003.14.9, , 00000000007.19.9, , , , , ] 2022-01-11T14:11:05.491 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:05.492+0000 7f41138a4500 0 bucket checkpoint complete 2022-01-11T14:11:05.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:05.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:05.508 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:05.557 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:05.558 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:05.617 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:05.619+0000 7f38881bb500 1 bucket sync caught up with source: 2022-01-11T14:11:05.617 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000010.23.5, , 00000000003.14.9, , 00000000007.19.9, , , , , ] 2022-01-11T14:11:05.617 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000010.23.5, , 00000000003.14.9, , 00000000007.19.9, , , , , ] 2022-01-11T14:11:05.617 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:05.619+0000 7f38881bb500 0 bucket checkpoint complete 2022-01-11T14:11:05.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:05.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:05.632 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:05.682 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:05.682 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:05.748 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:05.750+0000 7f436bf1a500 1 bucket sync caught up with source: 2022-01-11T14:11:05.748 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000010.23.5, , 00000000003.14.9, , 00000000007.19.9, , , , , ] 2022-01-11T14:11:05.749 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000010.23.5, , 00000000003.14.9, , 00000000007.19.9, , , , , ] 2022-01-11T14:11:05.749 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:05.750+0000 7f436bf1a500 0 bucket checkpoint complete 2022-01-11T14:11:05.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:05.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:05.769 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:05.819 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:05.819 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:05.881 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:05.883+0000 7f12e2de4500 1 bucket sync caught up with source: 2022-01-11T14:11:05.881 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000003.136.9, , 00000000010.244.5, , 00000000008.146.9, , , , , ] 2022-01-11T14:11:05.882 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000003.136.9, , 00000000010.244.5, , 00000000008.146.9, , , , , ] 2022-01-11T14:11:05.882 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:05.883+0000 7f12e2de4500 0 bucket checkpoint complete 2022-01-11T14:11:05.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:05.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:05.898 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:05.947 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:05.948 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:06.008 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:06.010+0000 7f9bbb3e8500 1 bucket sync caught up with source: 2022-01-11T14:11:06.008 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000003.136.9, , 00000000010.244.5, , 00000000008.146.9, , , , , ] 2022-01-11T14:11:06.008 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000003.136.9, , 00000000010.244.5, , 00000000008.146.9, , , , , ] 2022-01-11T14:11:06.009 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:06.010+0000 7f9bbb3e8500 0 bucket checkpoint complete 2022-01-11T14:11:06.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:06.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:06.030 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:06.079 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:06.080 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:06.136 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:06.139+0000 7f6249699500 1 bucket sync caught up with source: 2022-01-11T14:11:06.137 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000003.115.9, , 00000000003.306.9, , 00000000014.162.5, , , , , ] 2022-01-11T14:11:06.137 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000003.115.9, , 00000000003.306.9, , 00000000014.162.5, , , , , ] 2022-01-11T14:11:06.137 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:06.139+0000 7f6249699500 0 bucket checkpoint complete 2022-01-11T14:11:06.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:06.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:06.150 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:06.199 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:06.199 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:06.261 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:06.263+0000 7f8209f5c500 1 bucket sync caught up with source: 2022-01-11T14:11:06.262 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000003.115.9, , 00000000003.306.9, , 00000000014.162.5, , , , , ] 2022-01-11T14:11:06.262 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000003.115.9, , 00000000003.306.9, , 00000000014.162.5, , , , , ] 2022-01-11T14:11:06.262 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:06.263+0000 7f8209f5c500 0 bucket checkpoint complete 2022-01-11T14:11:06.281 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-17 zones={test-zone2, test-zone1} 2022-01-11T14:11:06.288 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:06.292 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:06.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:06.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:06.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:06.294 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:06.298 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:06.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:06.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:06.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:06.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:06.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:06.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:06.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:06.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:06.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:06.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:06.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:06.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:06.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:07.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:07.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:07.210 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-17 zones={test-zone2, test-zone1} 2022-01-11T14:11:07.211 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-17 zones={test-zone3, test-zone1} 2022-01-11T14:11:07.217 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:07.221 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:07.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:07.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:07.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:07.223 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:07.228 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:07.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:07.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:07.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:07.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:07.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:07.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:07.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:07.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:07.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:07.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:07.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:07.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:07.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:07.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:08.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:08.119 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-17 zones={test-zone3, test-zone1} 2022-01-11T14:11:08.119 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-17 zones={test-zone3, test-zone2} 2022-01-11T14:11:08.126 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:08.130 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:08.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:08.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:08.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:08.132 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:08.136 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:08.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:08.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:08.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:08.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:08.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:08.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:08.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:08.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:08.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:08.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:08.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:08.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:08.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:08.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:08.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:09.043 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-17 zones={test-zone3, test-zone2} 2022-01-11T14:11:09.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:09.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:09.045 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:09.100 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:09.100 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:09.167 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:09.168+0000 7f4c89b0e500 1 bucket sync caught up with source: 2022-01-11T14:11:09.167 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000010.31.5, , 00000000003.20.9, , 00000000013.28.5, , , , , ] 2022-01-11T14:11:09.167 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000010.31.5, , 00000000003.20.9, , 00000000013.28.5, , , , , ] 2022-01-11T14:11:09.167 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:09.168+0000 7f4c89b0e500 0 bucket checkpoint complete 2022-01-11T14:11:09.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:09.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:09.185 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:09.236 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:09.236 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:09.290 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:09.292+0000 7ff151221500 1 bucket sync caught up with source: 2022-01-11T14:11:09.290 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000010.31.5, , 00000000003.20.9, , 00000000013.28.5, , , , , ] 2022-01-11T14:11:09.290 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000010.31.5, , 00000000003.20.9, , 00000000013.28.5, , , , , ] 2022-01-11T14:11:09.291 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:09.292+0000 7ff151221500 0 bucket checkpoint complete 2022-01-11T14:11:09.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:09.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:09.308 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:09.358 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:09.358 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:09.420 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:09.422+0000 7f998b496500 1 bucket sync caught up with source: 2022-01-11T14:11:09.420 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000010.31.5, , 00000000003.20.9, , 00000000013.28.5, , , , , ] 2022-01-11T14:11:09.420 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000010.31.5, , 00000000003.20.9, , 00000000013.28.5, , , , , ] 2022-01-11T14:11:09.421 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:09.422+0000 7f998b496500 0 bucket checkpoint complete 2022-01-11T14:11:09.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:09.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:09.434 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:09.484 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:09.484 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:09.548 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:09.550+0000 7fe641613500 1 bucket sync caught up with source: 2022-01-11T14:11:09.548 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000007.338.5, , 00000000010.267.5, , 00000000013.56.5, , , , , ] 2022-01-11T14:11:09.548 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000007.338.5, , 00000000010.267.5, , 00000000013.56.5, , , , , ] 2022-01-11T14:11:09.549 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:09.550+0000 7fe641613500 0 bucket checkpoint complete 2022-01-11T14:11:09.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:09.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:09.565 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:09.614 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:09.614 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:09.663 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:09.666+0000 7fb0269c5500 1 bucket sync caught up with source: 2022-01-11T14:11:09.664 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000007.338.5, , 00000000010.267.5, , 00000000013.56.5, , , , , ] 2022-01-11T14:11:09.664 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000007.338.5, , 00000000010.267.5, , 00000000013.56.5, , , , , ] 2022-01-11T14:11:09.664 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:09.666+0000 7fb0269c5500 0 bucket checkpoint complete 2022-01-11T14:11:09.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:09.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:09.683 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:09.731 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:09.732 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:09.785 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:09.788+0000 7f642bfbd500 1 bucket sync caught up with source: 2022-01-11T14:11:09.786 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000004.431.9, , 00000000003.312.9, , 00000000015.69.5, , , , , ] 2022-01-11T14:11:09.786 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000004.431.9, , 00000000003.312.9, , 00000000015.69.5, , , , , ] 2022-01-11T14:11:09.786 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:09.788+0000 7f642bfbd500 0 bucket checkpoint complete 2022-01-11T14:11:09.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:09.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:09.804 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:09.857 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:09.858 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:09.910 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:09.912+0000 7f2689e95500 1 bucket sync caught up with source: 2022-01-11T14:11:09.911 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000004.431.9, , 00000000003.312.9, , 00000000015.69.5, , , , , ] 2022-01-11T14:11:09.911 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000004.431.9, , 00000000003.312.9, , 00000000015.69.5, , , , , ] 2022-01-11T14:11:09.911 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:09.912+0000 7f2689e95500 0 bucket checkpoint complete 2022-01-11T14:11:09.927 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-18 zones={test-zone2, test-zone1} 2022-01-11T14:11:09.934 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:09.937 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:09.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:09.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:09.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:09.939 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:09.943 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:09.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:09.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:09.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:09.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:09.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:10.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:10.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:10.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:10.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:10.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:10.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:10.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:10.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:10.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:10.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:10.862 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-18 zones={test-zone2, test-zone1} 2022-01-11T14:11:10.863 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-18 zones={test-zone3, test-zone1} 2022-01-11T14:11:10.869 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:10.873 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:10.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:10.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:10.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:10.875 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:10.879 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:10.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:10.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:10.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:10.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:10.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:10.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:11.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:11.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:11.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:11.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:11.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:11.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:11.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:11.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:11.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:11.786 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-18 zones={test-zone3, test-zone1} 2022-01-11T14:11:11.787 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-18 zones={test-zone3, test-zone2} 2022-01-11T14:11:11.793 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:11.797 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:11.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:11.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:11.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:11.799 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:11.803 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:11.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:11.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:11.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:11.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:11.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:11.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:11.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:11.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:12.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:12.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:12.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:12.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:12.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:12.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:12.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:12.723 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-18 zones={test-zone3, test-zone2} 2022-01-11T14:11:13.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:13.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:13.278 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:13.331 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:13.332 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:13.403 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:13.404+0000 7fab97386500 1 bucket sync caught up with source: 2022-01-11T14:11:13.403 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000011.34.5, , 00000000004.15.5, , 00000000009.26.5, , , , , ] 2022-01-11T14:11:13.403 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000011.34.5, , 00000000004.15.5, , 00000000009.26.5, , , , , ] 2022-01-11T14:11:13.403 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:13.405+0000 7fab97386500 0 bucket checkpoint complete 2022-01-11T14:11:13.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:13.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:13.416 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:13.468 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:13.469 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:13.537 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:13.539+0000 7fba1e308500 1 bucket sync caught up with source: 2022-01-11T14:11:13.537 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000011.34.5, , 00000000004.15.5, , 00000000009.26.5, , , , , ] 2022-01-11T14:11:13.538 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000011.34.5, , 00000000004.15.5, , 00000000009.26.5, , , , , ] 2022-01-11T14:11:13.538 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:13.539+0000 7fba1e308500 0 bucket checkpoint complete 2022-01-11T14:11:13.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:13.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:13.560 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:13.610 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:13.610 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:13.667 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:13.670+0000 7ff968f16500 1 bucket sync caught up with source: 2022-01-11T14:11:13.668 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000011.34.5, , 00000000004.15.5, , 00000000009.26.5, , , , , ] 2022-01-11T14:11:13.668 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000011.34.5, , 00000000004.15.5, , 00000000009.26.5, , , , , ] 2022-01-11T14:11:13.668 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:13.670+0000 7ff968f16500 0 bucket checkpoint complete 2022-01-11T14:11:13.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:13.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:13.691 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:13.742 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:13.742 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:13.804 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:13.806+0000 7ff9984a3500 1 bucket sync caught up with source: 2022-01-11T14:11:13.805 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000007.218.5, , 00000000011.286.5, , 00000000008.209.5, , , , , ] 2022-01-11T14:11:13.805 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000007.218.5, , 00000000011.286.5, , 00000000008.209.5, , , , , ] 2022-01-11T14:11:13.805 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:13.807+0000 7ff9984a3500 0 bucket checkpoint complete 2022-01-11T14:11:13.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:13.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:13.823 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:13.872 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:13.872 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:13.919 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:13.921+0000 7f7a0c2fb500 1 bucket sync caught up with source: 2022-01-11T14:11:13.920 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000007.218.5, , 00000000011.286.5, , 00000000008.209.5, , , , , ] 2022-01-11T14:11:13.920 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000007.218.5, , 00000000011.286.5, , 00000000008.209.5, , , , , ] 2022-01-11T14:11:13.920 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:13.921+0000 7f7a0c2fb500 0 bucket checkpoint complete 2022-01-11T14:11:13.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:13.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:13.944 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:13.992 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:13.993 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:14.057 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:14.059+0000 7f3cdced9500 1 bucket sync caught up with source: 2022-01-11T14:11:14.057 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000008.239.5, , 00000000004.303.5, , 00000000015.252.5, , , , , ] 2022-01-11T14:11:14.057 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000008.239.5, , 00000000004.303.5, , 00000000015.252.5, , , , , ] 2022-01-11T14:11:14.058 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:14.059+0000 7f3cdced9500 0 bucket checkpoint complete 2022-01-11T14:11:14.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:14.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:14.071 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:14.121 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:14.121 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:14.180 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:14.183+0000 7f6cb7135500 1 bucket sync caught up with source: 2022-01-11T14:11:14.181 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000008.239.5, , 00000000004.303.5, , 00000000015.252.5, , , , , ] 2022-01-11T14:11:14.181 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000008.239.5, , 00000000004.303.5, , 00000000015.252.5, , , , , ] 2022-01-11T14:11:14.181 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:14.183+0000 7f6cb7135500 0 bucket checkpoint complete 2022-01-11T14:11:14.203 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-16 zones={test-zone2, test-zone1} 2022-01-11T14:11:14.209 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:14.213 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:14.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:14.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:14.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:14.215 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:14.219 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:14.219 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:14.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:14.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:14.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:14.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:14.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:14.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:14.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:14.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:14.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:14.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:14.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:14.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:14.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:15.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:15.130 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-16 zones={test-zone2, test-zone1} 2022-01-11T14:11:15.131 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-16 zones={test-zone3, test-zone1} 2022-01-11T14:11:15.137 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:15.141 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:15.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:15.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:15.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:15.143 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:15.147 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:15.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:15.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:15.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:15.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:15.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:15.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:15.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:15.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:15.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:15.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:15.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:15.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:15.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:16.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:16.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:16.301 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-16 zones={test-zone3, test-zone1} 2022-01-11T14:11:16.302 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-16 zones={test-zone3, test-zone2} 2022-01-11T14:11:16.309 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:16.319 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:16.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:16.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:16.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:16.320 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:16.326 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:16.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:16.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:16.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:16.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:16.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:16.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:16.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:16.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:16.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:16.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:16.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:16.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:17.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:17.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:17.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:17.311 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-16 zones={test-zone3, test-zone2} 2022-01-11T14:11:17.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:17.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:17.313 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:17.367 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:17.368 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:17.438 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:17.440+0000 7fc1f68d2500 1 bucket sync caught up with source: 2022-01-11T14:11:17.439 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000011.29.5, , 00000000004.23.5, , 00000000008.22.5, , , , , ] 2022-01-11T14:11:17.439 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000011.29.5, , 00000000004.23.5, , 00000000008.22.5, , , , , ] 2022-01-11T14:11:17.439 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:17.440+0000 7fc1f68d2500 0 bucket checkpoint complete 2022-01-11T14:11:17.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:17.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:17.463 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:17.520 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:17.520 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:17.583 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:17.586+0000 7f711c282500 1 bucket sync caught up with source: 2022-01-11T14:11:17.584 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000011.29.5, , 00000000004.23.5, , 00000000008.22.5, , , , , ] 2022-01-11T14:11:17.584 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000011.29.5, , 00000000004.23.5, , 00000000008.22.5, , , , , ] 2022-01-11T14:11:17.584 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:17.586+0000 7f711c282500 0 bucket checkpoint complete 2022-01-11T14:11:17.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:17.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:17.606 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:17.659 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:17.660 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:17.714 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:17.716+0000 7f2f9b4f1500 1 bucket sync caught up with source: 2022-01-11T14:11:17.715 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000011.29.5, , 00000000004.23.5, , 00000000008.22.5, , , , , ] 2022-01-11T14:11:17.715 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000011.29.5, , 00000000004.23.5, , 00000000008.22.5, , , , , ] 2022-01-11T14:11:17.715 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:17.716+0000 7f2f9b4f1500 0 bucket checkpoint complete 2022-01-11T14:11:17.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:17.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:17.737 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:17.786 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:17.787 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:17.834 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:17.836+0000 7f7774400500 1 bucket sync caught up with source: 2022-01-11T14:11:17.835 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000004.139.5, , 00000000011.274.5, , 00000000009.149.5, , , , , ] 2022-01-11T14:11:17.835 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000004.139.5, , 00000000011.274.5, , 00000000009.149.5, , , , , ] 2022-01-11T14:11:17.835 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:17.836+0000 7f7774400500 0 bucket checkpoint complete 2022-01-11T14:11:17.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:17.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:17.850 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:17.898 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:17.899 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:17.958 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:17.960+0000 7fcc6e766500 1 bucket sync caught up with source: 2022-01-11T14:11:17.959 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000004.139.5, , 00000000011.274.5, , 00000000009.149.5, , , , , ] 2022-01-11T14:11:17.959 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000004.139.5, , 00000000011.274.5, , 00000000009.149.5, , , , , ] 2022-01-11T14:11:17.959 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:17.960+0000 7fcc6e766500 0 bucket checkpoint complete 2022-01-11T14:11:17.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:17.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:17.972 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:18.021 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:18.021 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:18.074 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:18.076+0000 7f64a39e4500 1 bucket sync caught up with source: 2022-01-11T14:11:18.074 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000004.118.5, , 00000000004.315.5, , 00000000015.165.5, , , , , ] 2022-01-11T14:11:18.075 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000004.118.5, , 00000000004.315.5, , 00000000015.165.5, , , , , ] 2022-01-11T14:11:18.075 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:18.076+0000 7f64a39e4500 0 bucket checkpoint complete 2022-01-11T14:11:18.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:18.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:18.099 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:18.150 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:18.151 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:18.210 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:18.212+0000 7f10dad4d500 1 bucket sync caught up with source: 2022-01-11T14:11:18.211 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000004.118.5, , 00000000004.315.5, , 00000000015.165.5, , , , , ] 2022-01-11T14:11:18.211 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000004.118.5, , 00000000004.315.5, , 00000000015.165.5, , , , , ] 2022-01-11T14:11:18.211 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:18.212+0000 7f10dad4d500 0 bucket checkpoint complete 2022-01-11T14:11:18.223 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-17 zones={test-zone2, test-zone1} 2022-01-11T14:11:18.230 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:18.235 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:18.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:18.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:18.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:18.237 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:18.240 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:18.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:18.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:18.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:18.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:18.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:18.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:18.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:18.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:18.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:18.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:18.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:18.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:18.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:18.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:19.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:19.178 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-17 zones={test-zone2, test-zone1} 2022-01-11T14:11:19.179 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-17 zones={test-zone3, test-zone1} 2022-01-11T14:11:19.185 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:19.189 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:19.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:19.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:19.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:19.191 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:19.195 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:19.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:19.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:19.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:19.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:19.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:19.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:19.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:19.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:19.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:19.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:19.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:19.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:19.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:19.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:20.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:20.106 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-17 zones={test-zone3, test-zone1} 2022-01-11T14:11:20.107 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-17 zones={test-zone3, test-zone2} 2022-01-11T14:11:20.113 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:20.117 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:20.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:20.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:20.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:20.119 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:20.124 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:20.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:20.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:20.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:20.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:20.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:20.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:20.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:20.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:20.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:20.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:20.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:20.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:20.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:20.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:20.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:21.059 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-17 zones={test-zone3, test-zone2} 2022-01-11T14:11:21.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:21.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:21.061 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:21.117 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:21.118 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:21.186 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:21.188+0000 7fe0c04bb500 1 bucket sync caught up with source: 2022-01-11T14:11:21.186 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000011.36.5, , 00000000004.25.5, , 00000000014.31.5, , , , , ] 2022-01-11T14:11:21.186 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000011.36.5, , 00000000004.25.5, , 00000000014.31.5, , , , , ] 2022-01-11T14:11:21.187 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:21.188+0000 7fe0c04bb500 0 bucket checkpoint complete 2022-01-11T14:11:21.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:21.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:21.210 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:21.260 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:21.261 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:21.322 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:21.324+0000 7f4bd2800500 1 bucket sync caught up with source: 2022-01-11T14:11:21.322 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000011.36.5, , 00000000004.25.5, , 00000000014.31.5, , , , , ] 2022-01-11T14:11:21.323 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000011.36.5, , 00000000004.25.5, , 00000000014.31.5, , , , , ] 2022-01-11T14:11:21.323 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:21.324+0000 7f4bd2800500 0 bucket checkpoint complete 2022-01-11T14:11:21.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:21.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:21.336 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:21.386 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:21.387 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:21.445 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:21.447+0000 7f63a5b3e500 1 bucket sync caught up with source: 2022-01-11T14:11:21.445 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000011.36.5, , 00000000004.25.5, , 00000000014.31.5, , , , , ] 2022-01-11T14:11:21.445 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000011.36.5, , 00000000004.25.5, , 00000000014.31.5, , , , , ] 2022-01-11T14:11:21.446 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:21.447+0000 7f63a5b3e500 0 bucket checkpoint complete 2022-01-11T14:11:21.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:21.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:21.458 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:21.508 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:21.509 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:21.556 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:21.558+0000 7fc16a0f4500 1 bucket sync caught up with source: 2022-01-11T14:11:21.557 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000008.342.5, , 00000000011.276.5, , 00000000014.59.5, , , , , ] 2022-01-11T14:11:21.557 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000008.342.5, , 00000000011.276.5, , 00000000014.59.5, , , , , ] 2022-01-11T14:11:21.557 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:21.558+0000 7fc16a0f4500 0 bucket checkpoint complete 2022-01-11T14:11:21.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:21.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:21.573 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:21.686 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:21.688+0000 7fd1b2ff3500 1 bucket sync caught up with source: 2022-01-11T14:11:21.686 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000008.342.5, , 00000000011.276.5, , 00000000014.59.5, , , , , ] 2022-01-11T14:11:21.687 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000008.342.5, , 00000000011.276.5, , 00000000014.59.5, , , , , ] 2022-01-11T14:11:21.687 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:21.688+0000 7fd1b2ff3500 0 bucket checkpoint complete 2022-01-11T14:11:21.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:21.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:21.704 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:21.755 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:21.755 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:21.824 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:21.826+0000 7f79cd32c500 1 bucket sync caught up with source: 2022-01-11T14:11:21.824 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, 00000000005.434.5, , 00000000004.317.5, , 00000000016.72.5, , , , , ] 2022-01-11T14:11:21.825 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, 00000000005.434.5, , 00000000004.317.5, , 00000000016.72.5, , , , , ] 2022-01-11T14:11:21.825 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:21.826+0000 7f79cd32c500 0 bucket checkpoint complete 2022-01-11T14:11:21.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:21.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:21.849 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:21.900 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:21.900 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:21.949 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:21.951+0000 7f4091d5f500 1 bucket sync caught up with source: 2022-01-11T14:11:21.949 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000005.434.5, , 00000000004.317.5, , 00000000016.72.5, , , , , ] 2022-01-11T14:11:21.950 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000005.434.5, , 00000000004.317.5, , 00000000016.72.5, , , , , ] 2022-01-11T14:11:21.950 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:21.951+0000 7f4091d5f500 0 bucket checkpoint complete 2022-01-11T14:11:21.963 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-18 zones={test-zone2, test-zone1} 2022-01-11T14:11:21.970 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:21.976 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:21.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:21.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:21.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:21.979 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:21.984 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:21.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:21.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:21.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:21.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:22.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:22.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:22.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:22.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:22.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:22.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:22.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:22.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:22.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:22.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:22.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:22.891 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-18 zones={test-zone2, test-zone1} 2022-01-11T14:11:22.892 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-18 zones={test-zone3, test-zone1} 2022-01-11T14:11:22.899 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:22.903 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:22.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:22.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:22.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:22.905 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:22.909 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:22.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:22.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:22.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:22.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:22.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:22.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:23.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:23.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:23.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:23.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:23.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:23.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:23.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:23.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:23.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:23.819 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-18 zones={test-zone3, test-zone1} 2022-01-11T14:11:23.819 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-18 zones={test-zone3, test-zone2} 2022-01-11T14:11:23.826 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:23.830 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:23.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:23.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:23.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:23.833 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:23.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2022-01-11T14:11:23.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2022-01-11T14:11:23.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2022-01-11T14:11:23.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:23.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:23.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:23.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:24.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:24.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2022-01-11T14:11:24.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:24.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:24.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2022-01-11T14:11:24.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:24.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:24.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2022-01-11T14:11:24.755 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-18 zones={test-zone3, test-zone2} 2022-01-11T14:11:24.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2022-01-11T14:11:24.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:11:24.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:11:24.788 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:11:24.834 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:24.834 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:24.895 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:11:24.915 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:11:24.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:11:24.916 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910284.788755_91.1", 2022-01-11T14:11:24.916 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:24.788755Z" 2022-01-11T14:11:24.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:11:24.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:11:24.917 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:11:24.918 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:11:24.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:11:24.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:11:24.919 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:11:24.919 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:11:24.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:11:24.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:11:24.921 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:11:24.921 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:11:24.921 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:11:24.922 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:11:24.924 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:11:24.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:11:24.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:11:24.926 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:11:24.975 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:24.975 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:25.043 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:11:25.044 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:11:25.045 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:11:25.045 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:11:25.046 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:11:25.046 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:11:25.047 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:25.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:25.048 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:11:25.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:11:25.049 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:11:25.049 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:11:25.049 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:11:25.050 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910284.766954_85.1", 2022-01-11T14:11:25.050 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:11:25.051 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:11:25.051 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:11:25.052 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:11:24.766954Z", 2022-01-11T14:11:25.052 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:25.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:25.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:25.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:11:25.054 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:11:25.054 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:11:25.055 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:11:25.055 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:11:25.055 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:11:25.056 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:11:25.056 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:11:25.057 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:11:25.057 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:25.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:25.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:25.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:11:25.059 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:11:25.059 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:11:25.060 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:11:25.060 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:11:25.061 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:11:25.061 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:11:25.062 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:11:25.062 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:11:25.062 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:25.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:25.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:25.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:11:25.064 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:11:25.065 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:11:25.065 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:11:25.065 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:11:25.066 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:11:25.066 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:11:25.067 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:11:25.067 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:11:25.068 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:25.068 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:25.068 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:25.069 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:11:25.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:25.070 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:11:25.070 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:11:25.071 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:11:25.071 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:25.071 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:11:25.074 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910284.788755_91.1 target=1_1641910284.766954_85.1 2022-01-11T14:11:30.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:11:30.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:11:30.076 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:11:30.125 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:30.125 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:30.193 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:11:30.194 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:11:30.194 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:11:30.195 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:11:30.195 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:11:30.195 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:11:30.195 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:30.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:30.196 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:11:30.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:11:30.196 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:11:30.197 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:11:30.197 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:11:30.197 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910284.788755_91.1", 2022-01-11T14:11:30.197 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:11:30.198 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:11:30.198 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:11:30.198 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:11:24.788755Z", 2022-01-11T14:11:30.198 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:30.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:30.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:30.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:11:30.199 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:11:30.200 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:11:30.200 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:11:30.200 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:11:30.200 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:11:30.201 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:11:30.201 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:11:30.201 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:11:30.201 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:30.202 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:30.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:30.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:11:30.202 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:11:30.203 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:11:30.203 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:11:30.203 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:11:30.203 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:11:30.204 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:11:30.204 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:11:30.204 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:11:30.204 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:30.205 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:30.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:30.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:11:30.205 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:11:30.206 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:11:30.206 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:11:30.206 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:11:30.206 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:11:30.207 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:11:30.207 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:11:30.208 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:11:30.208 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:30.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:30.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:30.208 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:11:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:11:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:11:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:11:30.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:30.210 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:11:30.211 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:11:30.212 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:11:30.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:11:30.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:11:30.213 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:11:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:30.319 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:11:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:11:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:11:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:11:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:11:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:11:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:11:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:11:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:11:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:11:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:11:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:11:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:11:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910284.788755_91.1", 2022-01-11T14:11:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:11:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:11:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:11:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:11:24.788755Z", 2022-01-11T14:11:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:11:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:11:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:11:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:11:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:11:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:11:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:11:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:11:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:11:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:11:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:11:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:11:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:11:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:11:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:11:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:11:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:11:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:11:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:11:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:11:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:11:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:11:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:11:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:11:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:11:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:11:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:11:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:11:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:11:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:11:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:11:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:11:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:11:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:11:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:11:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:11:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:11:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:11:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:11:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:11:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:11:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:11:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:11:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:11:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:11:30.334 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:11:30.335 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:11:30.336 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:11:30.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:11:30.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:11:30.337 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:11:30.381 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:30.382 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:30.451 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:11:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:11:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:11:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:11:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:11:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:11:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:11:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:11:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:11:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:11:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:11:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910284.788755_91.1", 2022-01-11T14:11:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:11:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:11:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:11:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:11:24.788755Z", 2022-01-11T14:11:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:11:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:11:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:11:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:11:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:11:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:11:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:11:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:11:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:11:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:11:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:11:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:11:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:11:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:11:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:11:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:11:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:11:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:11:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:11:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:11:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:11:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:11:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:11:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:11:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:11:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:11:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:11:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:11:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:11:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:11:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:11:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:11:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:11:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:11:30.467 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:11:30.468 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:11:30.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:30.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:30.516 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:30.566 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:30.566 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:30.616 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:30.619+0000 7f3420310500 1 waiting for incremental sync to catch up: 2022-01-11T14:11:30.617 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , , , ] 2022-01-11T14:11:30.617 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000002.15.10, , , , , ] 2022-01-11T14:11:35.617 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:35.620+0000 7f3420310500 1 bucket sync caught up with source: 2022-01-11T14:11:35.618 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.21.9, , , , , ] 2022-01-11T14:11:35.618 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000002.15.10, , , , , ] 2022-01-11T14:11:35.618 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:35.620+0000 7f3420310500 0 bucket checkpoint complete 2022-01-11T14:11:35.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:35.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:35.635 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:35.691 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:35.691 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:35.741 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:35.744+0000 7fa2ee46d500 1 bucket sync caught up with source: 2022-01-11T14:11:35.742 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , 00000000005.21.9, , , , , ] 2022-01-11T14:11:35.742 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , 00000000005.21.9, , , , , ] 2022-01-11T14:11:35.742 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:35.744+0000 7fa2ee46d500 0 bucket checkpoint complete 2022-01-11T14:11:35.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:35.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:35.759 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:35.815 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:35.816 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:35.877 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:35.880+0000 7efe24186500 1 bucket sync caught up with source: 2022-01-11T14:11:35.877 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.21.9, , , , , ] 2022-01-11T14:11:35.877 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000005.21.9, , , , , ] 2022-01-11T14:11:35.878 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:35.880+0000 7efe24186500 0 bucket checkpoint complete 2022-01-11T14:11:35.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:35.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:35.896 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:35.947 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:35.948 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:36.006 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:36.008+0000 7f918c57d500 1 bucket sync caught up with source: 2022-01-11T14:11:36.006 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , 00000000004.227.9, , , , , ] 2022-01-11T14:11:36.006 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , 00000000004.227.9, , , , , ] 2022-01-11T14:11:36.006 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:36.008+0000 7f918c57d500 0 bucket checkpoint complete 2022-01-11T14:11:36.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:36.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:36.029 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:36.078 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:36.078 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:36.136 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:36.138+0000 7fe889286500 1 bucket sync caught up with source: 2022-01-11T14:11:36.136 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , 00000000004.227.9, , , , , ] 2022-01-11T14:11:36.136 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , 00000000004.227.9, , , , , ] 2022-01-11T14:11:36.137 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:36.138+0000 7fe889286500 0 bucket checkpoint complete 2022-01-11T14:11:36.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:36.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:36.149 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:36.198 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:36.199 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:36.252 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:36.254+0000 7f509b63d500 1 bucket sync caught up with source: 2022-01-11T14:11:36.252 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , 00000000004.243.9, , , , , ] 2022-01-11T14:11:36.252 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , 00000000004.243.9, , , , , ] 2022-01-11T14:11:36.252 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:36.254+0000 7f509b63d500 0 bucket checkpoint complete 2022-01-11T14:11:36.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:36.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:36.266 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:36.315 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:36.316 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:36.362 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:36.365+0000 7f4ec8f8e500 1 bucket sync caught up with source: 2022-01-11T14:11:36.363 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000004.243.9, , , , , ] 2022-01-11T14:11:36.363 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000004.243.9, , , , , ] 2022-01-11T14:11:36.363 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:36.365+0000 7f4ec8f8e500 0 bucket checkpoint complete 2022-01-11T14:11:36.383 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-19 zones={test-zone2, test-zone1} 2022-01-11T14:11:36.390 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:36.435 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:36.435 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:36.439 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:36.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:36.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:36.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:36.614 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-19 zones={test-zone2, test-zone1} 2022-01-11T14:11:36.615 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-19 zones={test-zone3, test-zone1} 2022-01-11T14:11:36.620 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:36.667 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:36.668 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:36.672 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:36.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:36.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:36.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:36.846 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-19 zones={test-zone3, test-zone1} 2022-01-11T14:11:36.847 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-19 zones={test-zone3, test-zone2} 2022-01-11T14:11:36.853 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:36.903 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:36.904 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:36.907 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:36.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:36.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:37.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:37.083 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-19 zones={test-zone3, test-zone2} 2022-01-11T14:11:37.246 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:11:37.249+0000 7fc1e58d4700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2022-01-11T14:11:38.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:38.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:38.584 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:38.635 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:38.635 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:38.701 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:38.704+0000 7f3153606500 1 waiting for incremental sync to catch up: 2022-01-11T14:11:38.702 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.21.9, , , 00000000047.129.5, , ] 2022-01-11T14:11:38.702 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000005.21.9, , , 00000000052.140.9, , ] 2022-01-11T14:11:43.703 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:43.706+0000 7f3153606500 1 bucket sync caught up with source: 2022-01-11T14:11:43.703 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.21.9, , , 00000000053.142.9, , ] 2022-01-11T14:11:43.703 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000005.21.9, , , 00000000052.140.9, , ] 2022-01-11T14:11:43.704 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:43.706+0000 7f3153606500 0 bucket checkpoint complete 2022-01-11T14:11:43.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:43.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:43.727 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:43.783 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:43.784 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:43.835 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:43.837+0000 7f876e540500 1 bucket sync caught up with source: 2022-01-11T14:11:43.835 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , 00000000005.21.9, , , 00000000053.142.9, , ] 2022-01-11T14:11:43.835 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , 00000000005.21.9, , , 00000000053.142.9, , ] 2022-01-11T14:11:43.835 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:43.837+0000 7f876e540500 0 bucket checkpoint complete 2022-01-11T14:11:43.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:43.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:43.849 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:43.907 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:43.907 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:43.977 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:43.979+0000 7f1d26599500 1 bucket sync caught up with source: 2022-01-11T14:11:43.977 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.21.9, , , 00000000053.142.9, , ] 2022-01-11T14:11:43.977 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000005.21.9, , , 00000000053.142.9, , ] 2022-01-11T14:11:43.978 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:43.979+0000 7f1d26599500 0 bucket checkpoint complete 2022-01-11T14:11:43.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:43.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:43.992 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:44.043 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:44.044 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:44.104 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:44.106+0000 7feb7d68a500 1 bucket sync caught up with source: 2022-01-11T14:11:44.104 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , 00000000004.227.9, , , 00000000062.430.9, , ] 2022-01-11T14:11:44.105 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , 00000000004.227.9, , , 00000000062.430.9, , ] 2022-01-11T14:11:44.105 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:44.106+0000 7feb7d68a500 0 bucket checkpoint complete 2022-01-11T14:11:44.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:44.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:44.120 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:44.169 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:44.169 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:44.233 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:44.235+0000 7f4e729fc500 1 bucket sync caught up with source: 2022-01-11T14:11:44.234 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , 00000000004.227.9, , , 00000000062.430.9, , ] 2022-01-11T14:11:44.234 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , 00000000004.227.9, , , 00000000062.430.9, , ] 2022-01-11T14:11:44.234 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:44.235+0000 7f4e729fc500 0 bucket checkpoint complete 2022-01-11T14:11:44.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:44.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:44.257 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:44.306 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:44.307 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:44.363 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:44.365+0000 7f435395e500 1 bucket sync caught up with source: 2022-01-11T14:11:44.363 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , 00000000004.243.9, , , 00000000060.408.9, , ] 2022-01-11T14:11:44.363 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , 00000000004.243.9, , , 00000000060.408.9, , ] 2022-01-11T14:11:44.364 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:11:44.365+0000 7f435395e500 0 bucket checkpoint complete 2022-01-11T14:11:44.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:44.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:11:44.377 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:11:44.428 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:11:44.428 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:11:44.493 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:44.495+0000 7f0336287500 1 bucket sync caught up with source: 2022-01-11T14:11:44.493 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000004.243.9, , , 00000000060.408.9, , ] 2022-01-11T14:11:44.494 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000004.243.9, , , 00000000060.408.9, , ] 2022-01-11T14:11:44.494 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:11:44.495+0000 7f0336287500 0 bucket checkpoint complete 2022-01-11T14:11:44.510 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-19 zones={test-zone2, test-zone1} 2022-01-11T14:11:44.518 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:44.524 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:44.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.548 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:44.555 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:44.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:44.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:44.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:44.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:44.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:44.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:44.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:44.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:45.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:45.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:45.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:45.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:45.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:45.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:45.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:45.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:45.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:45.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:45.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:46.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:46.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:46.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:46.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:46.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:46.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:46.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:46.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:46.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:46.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:46.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:46.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:47.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:47.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:47.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:47.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:47.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:47.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:47.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:47.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:47.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:47.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:47.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:48.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:48.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:48.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:48.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:48.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:48.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:48.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:48.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:48.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:48.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:48.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:49.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:49.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:49.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:49.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:49.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:49.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:49.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:49.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:49.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:49.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:49.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:50.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:50.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:50.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:50.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:50.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:50.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:50.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:50.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:50.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:50.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:50.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:51.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:51.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:51.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:51.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:51.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:51.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:51.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:51.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:51.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:51.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:51.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:52.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:52.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:52.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:52.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:52.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:52.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:52.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:52.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:52.759 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-19 zones={test-zone2, test-zone1} 2022-01-11T14:11:52.760 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-19 zones={test-zone3, test-zone1} 2022-01-11T14:11:52.767 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:11:52.773 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:52.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.787 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:11:52.794 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:11:52.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:11:52.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:52.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:52.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:11:52.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:53.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:53.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:53.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:53.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:53.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:53.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:53.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:53.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:53.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:53.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:53.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:53.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:54.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:54.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:54.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:54.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:54.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:54.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:54.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:54.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:54.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:54.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:55.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:55.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:55.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:55.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:55.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:55.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:55.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:55.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:55.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:55.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:55.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:55.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:56.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:56.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:56.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:56.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:56.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:56.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:56.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:56.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:56.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:56.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:56.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:57.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:57.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:57.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:57.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:57.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:57.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:57.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:57.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:57.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:57.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:57.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:57.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:58.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:58.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:58.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:58.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:58.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:58.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:58.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:58.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:58.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:58.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:58.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:59.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:59.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:59.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:59.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:59.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:59.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:59.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:59.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:59.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:59.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:59.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:11:59.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:00.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:00.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:00.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:00.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:00.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:00.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:00.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:00.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:00.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:00.867 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-19 zones={test-zone3, test-zone1} 2022-01-11T14:12:00.867 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-19 zones={test-zone3, test-zone2} 2022-01-11T14:12:00.874 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:00.880 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:12:00.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.903 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:00.912 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2022-01-11T14:12:00.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:00.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:12:00.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:12:01.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2022-01-11T14:12:01.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:01.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:01.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:01.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:01.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:01.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:01.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:01.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:01.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:01.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:01.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:02.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:02.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:02.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:02.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:02.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:02.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:02.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:02.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:02.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:02.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:02.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:02.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:03.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:03.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:03.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:03.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:03.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:03.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:03.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:03.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:03.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:03.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:03.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:03.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:04.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:04.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:04.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:04.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:04.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:04.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:04.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:04.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:04.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:04.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:04.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:05.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:05.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:05.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:05.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:05.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:05.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:05.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:05.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:05.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:05.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:05.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:06.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:06.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:06.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:06.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:06.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:06.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:06.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:06.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:06.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:06.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:06.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:07.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:07.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:07.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:07.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:07.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:07.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:07.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:07.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:07.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:07.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:07.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:08.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:08.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:08.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:08.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:08.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:08.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:08.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:08.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:08.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:08.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:08.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:09.060 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-19 zones={test-zone3, test-zone2} 2022-01-11T14:12:09.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:09.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:09.062 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:09.110 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:09.110 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:09.190 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:12:09.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.191 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000004236", 2022-01-11T14:12:09.191 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:46.246260Z" 2022-01-11T14:12:09.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:09.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.193 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:09.193 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:09.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:09.194 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.194 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000002374", 2022-01-11T14:12:09.194 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:24.244923Z" 2022-01-11T14:12:09.194 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:09.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.195 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000001044", 2022-01-11T14:12:09.195 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:46.247553Z" 2022-01-11T14:12:09.196 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:09.196 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:12:09.197 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-01-11T14:12:09.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:09.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:09.199 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:09.251 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:09.252 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:09.322 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: "instance_id": 13937788665823961816 2022-01-11T14:12:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:09.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:09.325 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:09.325 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:09.325 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "incremental-sync", 2022-01-11T14:12:09.326 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000004236", 2022-01-11T14:12:09.326 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:09.326 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:09.326 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:09.327 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:11:46.246260Z" 2022-01-11T14:12:09.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "incremental-sync", 2022-01-11T14:12:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000" 2022-01-11T14:12:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "incremental-sync", 2022-01-11T14:12:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000002374", 2022-01-11T14:12:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:11:24.244923Z" 2022-01-11T14:12:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "incremental-sync", 2022-01-11T14:12:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000001044", 2022-01-11T14:12:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:11:46.247553Z" 2022-01-11T14:12:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 0, 2022-01-11T14:12:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 0 2022-01-11T14:12:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:09.338 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:09.340 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2022-01-11T14:12:09.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:09.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:09.341 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:09.385 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:09.386 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:09.470 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:12:09.470 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.471 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000004236", 2022-01-11T14:12:09.471 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:46.246260Z" 2022-01-11T14:12:09.471 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:09.471 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.472 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:09.472 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:09.472 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:09.472 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.472 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000002374", 2022-01-11T14:12:09.473 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:24.244923Z" 2022-01-11T14:12:09.473 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:09.473 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.473 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000001044", 2022-01-11T14:12:09.474 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:46.247553Z" 2022-01-11T14:12:09.474 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:09.474 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:12:09.475 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-01-11T14:12:09.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:09.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:09.477 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:09.560 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:09.624 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:12:09.624 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:12:09.624 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:12:09.625 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:12:09.625 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:12:09.625 INFO:teuthology.orchestra.run.smithi161.stdout: "instance_id": 1240183306993841828 2022-01-11T14:12:09.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:09.626 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:12:09.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.626 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:12:09.627 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:09.627 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:09.627 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000004236", 2022-01-11T14:12:09.627 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:09.627 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:09.628 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:09.628 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:11:46.246260Z" 2022-01-11T14:12:09.628 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:09.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:09.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.629 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:12:09.629 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:09.629 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:09.630 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:09.630 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:09.630 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:09.630 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:09.630 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000" 2022-01-11T14:12:09.631 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:09.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:09.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.631 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:12:09.632 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:09.632 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:09.632 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000002374", 2022-01-11T14:12:09.632 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:09.633 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:09.633 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:09.633 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:11:24.244923Z" 2022-01-11T14:12:09.633 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:09.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:09.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.634 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:12:09.634 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:09.634 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:09.635 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000001044", 2022-01-11T14:12:09.635 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:09.635 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:09.635 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:09.636 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:11:46.247553Z" 2022-01-11T14:12:09.636 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:09.636 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:09.636 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:12:09.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:09.637 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:12:09.637 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 0, 2022-01-11T14:12:09.637 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 0 2022-01-11T14:12:09.637 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:09.637 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:12:09.639 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2022-01-11T14:12:09.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:09.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:09.640 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:09.682 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:09.682 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:09.750 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:12:09.751 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.751 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000004236", 2022-01-11T14:12:09.751 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:46.246260Z" 2022-01-11T14:12:09.752 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:09.752 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.752 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:09.752 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:09.753 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:09.753 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.753 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000002374", 2022-01-11T14:12:09.753 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:24.244923Z" 2022-01-11T14:12:09.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:09.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:09.754 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000001044", 2022-01-11T14:12:09.754 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:46.247553Z" 2022-01-11T14:12:09.754 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:09.755 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:12:09.757 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-01-11T14:12:09.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:09.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:09.759 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:09.811 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:09.811 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:09.894 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:09.895 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:09.895 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:09.895 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:09.895 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:09.896 INFO:teuthology.orchestra.run.smithi172.stdout: "instance_id": 18361331074327310191 2022-01-11T14:12:09.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:09.896 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:09.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:09.897 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:09.897 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:09.897 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "incremental-sync", 2022-01-11T14:12:09.897 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000004236", 2022-01-11T14:12:09.898 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:09.898 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:09.898 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:09.898 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:11:46.246260Z" 2022-01-11T14:12:09.898 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:09.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:09.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:09.901 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:09.901 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:09.902 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "incremental-sync", 2022-01-11T14:12:09.902 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:09.903 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:09.903 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:09.903 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:09.903 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000" 2022-01-11T14:12:09.904 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:09.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:09.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:09.904 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:09.904 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:09.905 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "incremental-sync", 2022-01-11T14:12:09.905 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000002374", 2022-01-11T14:12:09.905 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:09.905 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:09.906 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:09.906 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:11:24.244923Z" 2022-01-11T14:12:09.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:09.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:09.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:09.907 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:09.907 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:09.908 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "incremental-sync", 2022-01-11T14:12:09.908 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000001044", 2022-01-11T14:12:09.908 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:09.909 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:09.909 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:09.909 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:11:46.247553Z" 2022-01-11T14:12:09.909 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:09.909 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:09.910 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:09.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:09.910 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:09.910 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 0, 2022-01-11T14:12:09.911 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 0 2022-01-11T14:12:09.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:09.911 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:09.912 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2022-01-11T14:12:09.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:09.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:09.913 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:09.956 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:09.957 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:10.036 INFO:teuthology.orchestra.run.smithi172.stdout:[ 2022-01-11T14:12:10.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:10.036 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000004369", 2022-01-11T14:12:10.037 INFO:teuthology.orchestra.run.smithi172.stdout: "last_update": "2022-01-11T14:11:30.965933Z" 2022-01-11T14:12:10.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:10.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:10.037 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:10.038 INFO:teuthology.orchestra.run.smithi172.stdout: "last_update": "0.000000" 2022-01-11T14:12:10.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:10.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:10.038 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000002374", 2022-01-11T14:12:10.039 INFO:teuthology.orchestra.run.smithi172.stdout: "last_update": "2022-01-11T14:11:30.967759Z" 2022-01-11T14:12:10.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:10.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:10.039 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000001044", 2022-01-11T14:12:10.040 INFO:teuthology.orchestra.run.smithi172.stdout: "last_update": "2022-01-11T14:11:52.969231Z" 2022-01-11T14:12:10.040 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:10.040 INFO:teuthology.orchestra.run.smithi172.stdout:] 2022-01-11T14:12:10.041 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-01-11T14:12:10.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.043 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:10.088 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:10.089 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:10.161 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:12:10.162 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:12:10.162 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:12:10.162 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:12:10.162 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:12:10.163 INFO:teuthology.orchestra.run.smithi161.stdout: "instance_id": 2408378516117901460 2022-01-11T14:12:10.163 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.163 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:12:10.163 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.164 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:12:10.164 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:10.164 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.164 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000004369", 2022-01-11T14:12:10.165 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:10.165 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:10.165 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:10.165 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:11:30.965933Z" 2022-01-11T14:12:10.165 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.166 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.166 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:12:10.166 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:10.167 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.167 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:10.168 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:10.168 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:10.168 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:10.169 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000" 2022-01-11T14:12:10.169 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.169 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.169 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:12:10.170 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:10.170 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.170 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000002374", 2022-01-11T14:12:10.170 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:10.171 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:10.171 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:10.171 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:11:30.967759Z" 2022-01-11T14:12:10.171 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.172 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:12:10.172 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:10.173 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.173 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000001044", 2022-01-11T14:12:10.173 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:10.173 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:10.173 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:10.174 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:11:52.969231Z" 2022-01-11T14:12:10.174 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.174 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.174 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:12:10.175 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.175 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:12:10.175 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 0, 2022-01-11T14:12:10.175 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 0 2022-01-11T14:12:10.176 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.176 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:12:10.177 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2022-01-11T14:12:10.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.179 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:10.220 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:10.221 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:10.286 INFO:teuthology.orchestra.run.smithi172.stdout:[ 2022-01-11T14:12:10.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:10.287 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000004369", 2022-01-11T14:12:10.287 INFO:teuthology.orchestra.run.smithi172.stdout: "last_update": "2022-01-11T14:11:30.965933Z" 2022-01-11T14:12:10.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:10.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:10.288 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:10.288 INFO:teuthology.orchestra.run.smithi172.stdout: "last_update": "0.000000" 2022-01-11T14:12:10.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:10.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:10.289 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000002374", 2022-01-11T14:12:10.289 INFO:teuthology.orchestra.run.smithi172.stdout: "last_update": "2022-01-11T14:11:30.967759Z" 2022-01-11T14:12:10.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:10.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:10.290 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000001044", 2022-01-11T14:12:10.290 INFO:teuthology.orchestra.run.smithi172.stdout: "last_update": "2022-01-11T14:11:52.969231Z" 2022-01-11T14:12:10.290 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:10.291 INFO:teuthology.orchestra.run.smithi172.stdout:] 2022-01-11T14:12:10.292 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-01-11T14:12:10.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.294 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:10.339 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:10.339 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:10.400 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:12:10.401 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:12:10.401 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:12:10.401 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:12:10.402 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:12:10.402 INFO:teuthology.orchestra.run.smithi161.stdout: "instance_id": 11222840623676102568 2022-01-11T14:12:10.402 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.402 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:12:10.403 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.403 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:12:10.403 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:10.403 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.403 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000004369", 2022-01-11T14:12:10.404 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:10.404 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:10.404 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:10.405 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:11:30.965933Z" 2022-01-11T14:12:10.405 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.405 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.406 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.406 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:12:10.406 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:10.406 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.407 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:10.407 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:10.407 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:10.407 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:10.408 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000" 2022-01-11T14:12:10.408 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.408 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.408 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.409 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:12:10.409 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:10.409 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.409 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000002374", 2022-01-11T14:12:10.409 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:10.410 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:10.410 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:10.410 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:11:30.967759Z" 2022-01-11T14:12:10.410 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.411 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.411 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:12:10.411 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:10.412 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.412 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000001044", 2022-01-11T14:12:10.412 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:10.412 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:10.413 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:10.413 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:11:52.969231Z" 2022-01-11T14:12:10.413 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.413 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.413 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:12:10.414 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.414 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:12:10.414 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 0, 2022-01-11T14:12:10.414 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 0 2022-01-11T14:12:10.415 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.415 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:12:10.416 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2022-01-11T14:12:10.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.418 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:10.460 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:10.461 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:10.515 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:12:10.516 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.516 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000004236", 2022-01-11T14:12:10.516 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:46.252257Z" 2022-01-11T14:12:10.516 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.517 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.517 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:10.517 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:10.517 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.518 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.518 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000002374", 2022-01-11T14:12:10.518 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:24.251002Z" 2022-01-11T14:12:10.518 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.519 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.519 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000001044", 2022-01-11T14:12:10.519 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:46.256678Z" 2022-01-11T14:12:10.519 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.520 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:12:10.521 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-01-11T14:12:10.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.523 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:10.607 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:10.607 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:10.681 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:12:10.681 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:12:10.682 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:12:10.682 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:12:10.682 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:12:10.682 INFO:teuthology.orchestra.run.smithi161.stdout: "instance_id": 12492509543539040497 2022-01-11T14:12:10.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.683 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:12:10.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.683 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:12:10.684 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:10.684 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.684 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000004236", 2022-01-11T14:12:10.684 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:10.685 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:10.685 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:10.685 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:11:46.252257Z" 2022-01-11T14:12:10.685 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.686 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:12:10.686 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:10.686 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.687 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:10.687 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:10.687 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:10.687 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:10.687 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000" 2022-01-11T14:12:10.688 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.688 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:12:10.689 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:10.689 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.689 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000002374", 2022-01-11T14:12:10.689 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:10.690 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:10.690 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:10.690 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:11:24.251002Z" 2022-01-11T14:12:10.690 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.691 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.691 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:12:10.691 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:10.691 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.692 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000001044", 2022-01-11T14:12:10.692 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:10.692 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:10.692 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:10.693 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:11:46.256678Z" 2022-01-11T14:12:10.693 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.693 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.693 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:12:10.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.694 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:12:10.694 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 0, 2022-01-11T14:12:10.694 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 0 2022-01-11T14:12:10.694 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.695 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:12:10.696 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2022-01-11T14:12:10.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.698 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:10.740 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:10.740 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:10.814 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:12:10.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.815 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000004236", 2022-01-11T14:12:10.815 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:46.252257Z" 2022-01-11T14:12:10.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.816 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:10.816 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:10.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.817 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.817 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000002374", 2022-01-11T14:12:10.817 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:24.251002Z" 2022-01-11T14:12:10.817 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:10.818 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:10.818 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "00000000000000000000:00000000000000001044", 2022-01-11T14:12:10.818 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:11:46.256678Z" 2022-01-11T14:12:10.818 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:10.818 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:12:10.820 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-01-11T14:12:10.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.821 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:10.866 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:10.867 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:10.937 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:10.938 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:10.938 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:10.938 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:10.939 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:10.939 INFO:teuthology.orchestra.run.smithi172.stdout: "instance_id": 7844662207366393295 2022-01-11T14:12:10.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:10.939 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:10.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:10.940 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:10.940 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:10.940 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.941 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000004236", 2022-01-11T14:12:10.941 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:10.941 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:10.941 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:10.941 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:11:46.252257Z" 2022-01-11T14:12:10.942 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:10.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:10.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:10.942 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:10.942 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:10.943 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.943 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:10.943 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:10.943 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:10.944 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:10.944 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000" 2022-01-11T14:12:10.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:10.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:10.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:10.946 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:10.946 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:10.946 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.947 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000002374", 2022-01-11T14:12:10.947 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:10.947 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:10.948 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:10.948 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:11:24.251002Z" 2022-01-11T14:12:10.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:10.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:10.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:10.949 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:10.949 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:10.949 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "incremental-sync", 2022-01-11T14:12:10.950 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "00000000000000000000:00000000000000001044", 2022-01-11T14:12:10.950 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:10.950 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:10.950 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:10.950 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:11:46.256678Z" 2022-01-11T14:12:10.951 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:10.951 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:10.952 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:10.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:10.952 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:10.953 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 0, 2022-01-11T14:12:10.954 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 0 2022-01-11T14:12:10.954 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:10.955 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:10.964 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2022-01-11T14:12:10.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2022-01-11T14:12:10.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:10.991 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:11.034 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:11.034 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:11.079 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:12:11.095 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:12:11.095 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:11.096 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910330.989892_629.1", 2022-01-11T14:12:11.096 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:12:10.989892Z" 2022-01-11T14:12:11.096 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:11.096 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:11.097 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:11.097 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:12:11.097 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:11.097 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:11.097 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:11.098 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:11.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:11.098 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:11.098 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:11.099 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:11.099 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:11.099 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:12:11.100 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:12:11.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:11.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:11.103 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:11.152 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:11.153 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:11.199 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:11.199 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910330.980375_626.1", 2022-01-11T14:12:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:12:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:10.980375Z", 2022-01-11T14:12:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:12:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:12:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:11.214 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:11.216 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910330.989892_629.1 target=1_1641910330.980375_626.1 2022-01-11T14:12:16.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:16.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:16.219 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:16.268 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:16.269 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:16.352 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:16.352 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:16.353 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:16.354 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910330.989892_629.1", 2022-01-11T14:12:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:16.355 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:12:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:10.989892Z", 2022-01-11T14:12:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:16.357 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:16.358 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:16.359 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:16.362 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:16.363 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:16.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:16.365 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:16.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:16.368 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:12:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:12:16.369 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:16.369 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:16.371 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:12:16.372 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:12:16.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:16.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:16.373 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:16.424 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:16.425 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:16.506 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:12:16.507 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:12:16.507 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:12:16.507 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:12:16.507 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:12:16.508 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:16.508 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:16.508 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:16.508 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:12:16.508 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:16.509 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:12:16.509 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:16.509 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:16.509 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910330.989892_629.1", 2022-01-11T14:12:16.510 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:16.510 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:12:16.510 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:16.510 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:12:10.989892Z", 2022-01-11T14:12:16.510 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:16.511 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:16.511 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:16.511 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:16.511 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:12:16.512 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:16.512 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:16.512 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:16.512 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:16.512 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:16.513 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:16.513 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:16.513 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:16.513 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:16.514 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:16.514 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:16.514 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:12:16.514 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:16.514 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:16.515 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:16.515 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:16.515 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:16.515 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:16.516 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:12:16.516 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:16.516 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:16.516 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:16.517 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:16.517 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:12:16.517 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:16.517 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:16.517 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:16.518 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:16.518 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:16.518 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:16.518 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:12:16.519 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:16.519 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:16.519 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:16.519 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:12:16.520 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:16.520 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:12:16.520 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:12:16.520 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:12:16.520 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:16.521 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:12:16.522 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:12:16.523 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:12:16.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:16.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:16.524 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:16.570 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:16.570 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:16.634 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:16.634 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:16.636 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:16.638 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910330.989892_629.1", 2022-01-11T14:12:16.639 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:12:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:16.640 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:10.989892Z", 2022-01-11T14:12:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:16.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:16.642 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:16.643 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:16.644 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:16.645 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:16.646 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:16.647 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:16.648 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:16.649 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:16.650 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:12:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:12:16.651 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:16.651 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:16.653 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:12:16.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:16.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:16.705 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:16.757 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:16.757 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:16.804 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:16.808+0000 7f96ba794500 1 waiting for incremental sync to catch up: 2022-01-11T14:12:16.805 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , , , ] 2022-01-11T14:12:16.805 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.12.5, , ] 2022-01-11T14:12:21.806 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:21.809+0000 7f96ba794500 1 bucket sync caught up with source: 2022-01-11T14:12:21.806 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-01-11T14:12:21.807 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.12.5, , ] 2022-01-11T14:12:21.807 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:21.809+0000 7f96ba794500 0 bucket checkpoint complete 2022-01-11T14:12:21.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:21.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:21.829 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:21.886 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:21.887 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:21.956 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:21.958+0000 7f6f2e216500 1 bucket sync caught up with source: 2022-01-11T14:12:21.956 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-01-11T14:12:21.957 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000001.12.5, , ] 2022-01-11T14:12:21.957 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:21.958+0000 7f6f2e216500 0 bucket checkpoint complete 2022-01-11T14:12:21.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:21.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:21.979 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:22.037 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:22.038 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:22.109 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:22.112+0000 7f755422c500 1 bucket sync caught up with source: 2022-01-11T14:12:22.109 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-01-11T14:12:22.110 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.12.5, , ] 2022-01-11T14:12:22.110 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:22.112+0000 7f755422c500 0 bucket checkpoint complete 2022-01-11T14:12:22.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:22.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:22.125 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:22.175 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:22.175 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:22.228+0000 7f48dfaed500 1 bucket sync caught up with source: 2022-01-11T14:12:22.226 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000001.277.5, , ] 2022-01-11T14:12:22.227 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000001.277.5, , ] 2022-01-11T14:12:22.227 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:22.228+0000 7f48dfaed500 0 bucket checkpoint complete 2022-01-11T14:12:22.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:22.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:22.240 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:22.291 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:22.291 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:22.346 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:22.348+0000 7f042e66f500 1 bucket sync caught up with source: 2022-01-11T14:12:22.347 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000001.277.5, , ] 2022-01-11T14:12:22.347 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000001.277.5, , ] 2022-01-11T14:12:22.347 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:22.348+0000 7f042e66f500 0 bucket checkpoint complete 2022-01-11T14:12:22.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:22.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:22.362 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:22.412 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:22.413 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:22.457 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:22.459+0000 7fefc5e82500 1 bucket sync caught up with source: 2022-01-11T14:12:22.457 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000001.255.5, , ] 2022-01-11T14:12:22.458 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000001.255.5, , ] 2022-01-11T14:12:22.458 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:22.459+0000 7fefc5e82500 0 bucket checkpoint complete 2022-01-11T14:12:22.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:22.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:22.475 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:22.526 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:22.527 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:22.575 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:22.577+0000 7f2e055aa500 1 bucket sync caught up with source: 2022-01-11T14:12:22.575 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.255.5, , ] 2022-01-11T14:12:22.575 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.255.5, , ] 2022-01-11T14:12:22.575 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:22.577+0000 7f2e055aa500 0 bucket checkpoint complete 2022-01-11T14:12:22.587 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-20 zones={test-zone2, test-zone1} 2022-01-11T14:12:22.594 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:22.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:22.640 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:22.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:22.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:22.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:22.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:22.822 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-20 zones={test-zone2, test-zone1} 2022-01-11T14:12:22.823 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-20 zones={test-zone3, test-zone1} 2022-01-11T14:12:22.829 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:22.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:22.880 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:22.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:22.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:22.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:23.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:23.062 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-20 zones={test-zone3, test-zone1} 2022-01-11T14:12:23.063 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-20 zones={test-zone3, test-zone2} 2022-01-11T14:12:23.070 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:23.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:23.115 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:23.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:23.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:23.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:23.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:23.343 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-20 zones={test-zone3, test-zone2} 2022-01-11T14:12:23.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:23.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:23.354 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:23.395 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:23.396 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:23.440 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:12:23.453 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:12:23.454 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:23.454 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910343.354590_631.1", 2022-01-11T14:12:23.454 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:12:23.354590Z" 2022-01-11T14:12:23.454 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:23.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:23.455 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:23.455 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:12:23.455 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:23.456 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:23.456 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:23.456 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:23.456 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:23.457 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:23.457 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:23.457 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:23.457 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:23.457 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:12:23.459 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:12:23.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:23.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:23.460 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:23.504 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:23.505 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:23.587 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:23.588 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:23.588 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:23.588 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:23.589 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:23.589 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:23.589 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:23.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:23.590 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:23.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:23.590 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:23.590 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:23.591 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:23.591 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910330.989892_629.1", 2022-01-11T14:12:23.591 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:23.591 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:12:23.592 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:23.592 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:10.989892Z", 2022-01-11T14:12:23.592 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:23.592 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:23.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:23.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:23.593 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:23.593 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:23.594 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:23.594 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:23.594 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:23.594 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:23.595 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:23.595 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:23.595 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:23.595 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:23.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:23.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:23.596 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:23.596 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:23.596 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:23.597 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:23.597 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:23.597 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:23.598 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:23.598 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:23.598 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:23.598 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:23.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:23.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:23.599 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:23.599 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:23.599 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:23.600 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:23.600 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:23.600 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:23.600 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:23.601 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:23.601 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:23.601 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:23.601 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:23.601 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:23.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:23.602 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:23.602 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:12:23.602 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:12:23.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:23.603 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:23.604 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910343.354590_631.1 target=1_1641910330.989892_629.1 2022-01-11T14:12:28.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:28.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:28.608 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:28.659 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:28.659 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:28.721 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:28.721 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:28.721 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:28.722 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:28.722 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:28.722 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:28.722 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:28.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:28.723 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:28.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:28.723 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:28.724 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:28.724 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:28.724 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910343.354590_631.1", 2022-01-11T14:12:28.724 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:28.725 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:12:28.725 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:28.725 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:23.354590Z", 2022-01-11T14:12:28.725 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:28.726 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:28.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:28.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:28.726 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:28.727 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:28.727 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:28.727 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:28.727 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:28.727 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:28.728 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:28.728 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:28.728 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:28.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:28.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:28.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:28.729 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:28.730 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:28.730 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:28.730 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:28.730 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:28.731 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:28.731 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:28.731 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:28.731 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:28.732 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:28.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:28.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:28.732 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:28.733 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:28.733 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:28.733 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:28.733 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:28.734 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:28.734 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:28.734 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:28.734 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:28.734 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:28.735 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:28.735 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:28.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:28.735 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:28.736 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:12:28.736 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:12:28.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:28.736 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:28.737 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:12:28.738 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:12:28.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:28.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:28.740 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:28.789 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:28.790 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:28.844 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:12:28.845 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:12:28.845 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:12:28.845 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:12:28.846 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:12:28.846 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:28.846 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:28.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:28.846 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:12:28.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:28.847 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:12:28.847 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:28.847 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:28.848 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910343.354590_631.1", 2022-01-11T14:12:28.848 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:28.848 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:12:28.848 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:28.849 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:12:23.354590Z", 2022-01-11T14:12:28.849 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:28.849 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:28.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:28.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:28.850 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:12:28.850 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:28.850 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:28.850 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:28.851 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:28.851 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:28.851 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:28.851 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:28.852 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:28.852 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:28.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:28.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:28.853 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:12:28.853 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:28.853 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:28.853 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:28.853 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:28.854 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:28.854 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:28.854 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:12:28.854 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:28.855 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:28.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:28.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:28.855 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:12:28.855 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:28.856 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:28.856 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:28.856 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:28.856 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:28.857 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:28.857 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:12:28.857 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:28.857 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:28.857 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:28.858 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:12:28.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:28.858 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:12:28.858 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:12:28.859 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:12:28.859 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:28.859 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:12:28.860 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:12:28.861 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:12:28.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:28.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:28.863 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:28.907 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:28.907 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:28.968 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:28.969 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:28.969 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:28.969 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:28.969 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:28.970 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:28.970 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:28.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:28.970 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:28.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:28.971 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:28.971 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:28.971 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:28.972 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910343.354590_631.1", 2022-01-11T14:12:28.972 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:28.972 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:12:28.972 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:28.973 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:23.354590Z", 2022-01-11T14:12:28.973 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:28.973 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:28.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:28.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:28.974 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:28.974 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:28.974 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:28.975 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:28.975 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:28.975 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:28.975 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:28.975 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:28.976 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:28.976 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:28.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:28.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:28.977 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:28.977 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:28.977 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:28.977 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:28.977 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:28.978 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:28.978 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:28.978 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:28.978 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:28.979 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:28.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:28.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:28.979 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:28.980 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:28.980 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:28.980 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:28.980 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:28.980 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:28.981 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:28.981 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:28.981 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:28.981 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:28.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:28.982 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:28.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:28.982 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:28.983 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:12:28.983 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:12:28.983 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:28.983 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:28.984 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:12:29.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:29.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:29.037 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:29.086 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:29.087 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:29.147 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:29.148+0000 7f2fb3e67500 1 waiting for incremental sync to catch up: 2022-01-11T14:12:29.147 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.12.5, , ] 2022-01-11T14:12:29.148 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-01-11T14:12:34.148 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:34.149+0000 7f2fb3e67500 1 bucket sync caught up with source: 2022-01-11T14:12:34.148 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2022-01-11T14:12:34.148 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-01-11T14:12:34.149 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:34.149+0000 7f2fb3e67500 0 bucket checkpoint complete 2022-01-11T14:12:34.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:34.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:34.166 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:34.222 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:34.223 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:34.281 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:34.283+0000 7ff28b8b5500 1 bucket sync caught up with source: 2022-01-11T14:12:34.281 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2022-01-11T14:12:34.282 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-01-11T14:12:34.282 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:34.283+0000 7ff28b8b5500 0 bucket checkpoint complete 2022-01-11T14:12:34.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:34.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:34.306 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:34.362 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:34.363 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:34.425 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:34.427+0000 7f0055c67500 1 bucket sync caught up with source: 2022-01-11T14:12:34.426 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2022-01-11T14:12:34.426 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2022-01-11T14:12:34.426 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:34.427+0000 7f0055c67500 0 bucket checkpoint complete 2022-01-11T14:12:34.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:34.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:34.447 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:34.498 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:34.499 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:34.545 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:34.547+0000 7fcf7e40f500 1 bucket sync caught up with source: 2022-01-11T14:12:34.545 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000004.302.9, , ] 2022-01-11T14:12:34.546 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000004.302.9, , ] 2022-01-11T14:12:34.546 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:34.547+0000 7fcf7e40f500 0 bucket checkpoint complete 2022-01-11T14:12:34.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:34.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:34.559 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:34.609 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:34.609 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:34.681+0000 7fb9d589c500 1 bucket sync caught up with source: 2022-01-11T14:12:34.679 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000004.302.9, , ] 2022-01-11T14:12:34.680 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000004.302.9, , ] 2022-01-11T14:12:34.680 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:34.681+0000 7fb9d589c500 0 bucket checkpoint complete 2022-01-11T14:12:34.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:34.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:34.698 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:34.748 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:34.749 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:34.815 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:34.817+0000 7f2896fae500 1 bucket sync caught up with source: 2022-01-11T14:12:34.815 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000004.270.9, , ] 2022-01-11T14:12:34.815 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000004.270.9, , ] 2022-01-11T14:12:34.816 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:34.817+0000 7f2896fae500 0 bucket checkpoint complete 2022-01-11T14:12:34.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:34.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:34.839 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:34.896 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:34.897 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:34.963 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:34.965+0000 7f2d9b181500 1 bucket sync caught up with source: 2022-01-11T14:12:34.964 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000004.270.9, , ] 2022-01-11T14:12:34.964 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000004.270.9, , ] 2022-01-11T14:12:34.964 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:34.965+0000 7f2d9b181500 0 bucket checkpoint complete 2022-01-11T14:12:34.981 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-20 zones={test-zone2, test-zone1} 2022-01-11T14:12:34.989 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:35.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:35.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:35.040 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:35.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:35.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:35.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:35.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:35.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:35.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:35.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:35.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:35.362 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-20 zones={test-zone2, test-zone1} 2022-01-11T14:12:35.363 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-20 zones={test-zone3, test-zone1} 2022-01-11T14:12:35.369 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:35.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:35.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:35.417 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:35.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:35.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:35.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:35.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:35.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:35.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:35.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:35.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:35.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-20 zones={test-zone3, test-zone1} 2022-01-11T14:12:35.763 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-20 zones={test-zone3, test-zone2} 2022-01-11T14:12:35.770 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:35.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:35.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:35.816 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:35.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:35.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:35.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:35.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:35.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:36.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:36.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:36.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:36.167 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-20 zones={test-zone3, test-zone2} 2022-01-11T14:12:36.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:36.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:36.177 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:36.220 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:36.221 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:36.276 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:12:36.297 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:12:36.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:36.298 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910356.178033_633.1", 2022-01-11T14:12:36.298 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:12:36.178033Z" 2022-01-11T14:12:36.299 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:36.299 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:36.299 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:36.299 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:12:36.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:36.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:36.300 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:36.300 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:36.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:36.301 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:36.301 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:36.302 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:36.302 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:36.302 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:12:36.304 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:12:36.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:36.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:36.305 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:36.351 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:36.352 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:36.413 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:36.414 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:36.414 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:36.414 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:36.415 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:36.415 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:36.415 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:36.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:36.416 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:36.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:36.416 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:36.417 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:36.417 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:36.417 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910356.175197_632.1", 2022-01-11T14:12:36.417 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:36.417 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:12:36.418 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:36.418 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:36.175197Z", 2022-01-11T14:12:36.418 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:36.418 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:36.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:36.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:36.419 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:36.419 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:36.420 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:36.420 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:36.420 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:36.420 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:36.421 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:36.421 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:36.421 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:36.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:36.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:36.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:36.422 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:36.422 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:36.423 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:36.423 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:36.423 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:36.423 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:36.424 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:36.424 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:36.424 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:36.425 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:36.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:36.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:36.426 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:36.427 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:36.427 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:36.427 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:36.428 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:36.428 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:36.429 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:36.429 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:36.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:36.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:36.432 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:36.432 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:12:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:12:36.433 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:36.434 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:36.436 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910356.178033_633.1 target=1_1641910356.175197_632.1 2022-01-11T14:12:41.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:41.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:41.440 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:41.490 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:41.491 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:41.563 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:41.563 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:41.564 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:41.564 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:41.564 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:41.564 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:41.565 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:41.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:41.565 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:41.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:41.566 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:41.566 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:41.566 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:41.566 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910356.178033_633.1", 2022-01-11T14:12:41.567 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:41.567 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:12:41.567 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:41.567 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:36.178033Z", 2022-01-11T14:12:41.567 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:41.568 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:41.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:41.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:41.568 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:41.568 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:41.569 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:41.569 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:41.569 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:41.569 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:41.569 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:41.570 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:41.570 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:41.570 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:41.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:41.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:41.571 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:41.571 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:41.571 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:41.571 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:41.572 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:41.572 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:41.572 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:41.572 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:41.572 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:41.573 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:41.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:41.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:41.573 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:41.574 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:41.574 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:41.574 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:41.574 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:41.574 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:41.575 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:41.575 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:41.575 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:41.575 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:12:41.577 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:12:41.577 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:41.577 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:41.578 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:12:41.579 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:12:41.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:41.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:41.580 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:41.629 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:41.630 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:41.690 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:12:41.691 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:12:41.691 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:12:41.691 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:12:41.691 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:12:41.692 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:41.692 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:41.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:41.692 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:12:41.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:41.693 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:12:41.693 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:41.693 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:41.694 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910356.178033_633.1", 2022-01-11T14:12:41.694 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:41.694 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:12:41.694 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:41.695 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:12:36.178033Z", 2022-01-11T14:12:41.695 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:41.695 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:41.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:41.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:41.696 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:12:41.696 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:41.696 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:41.697 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:41.697 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:41.697 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:41.697 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:41.697 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:41.698 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:41.698 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:41.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:41.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:41.699 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:12:41.699 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:41.699 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:41.699 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:41.700 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:41.700 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:41.700 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:41.700 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:12:41.700 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:41.701 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:41.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:41.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:41.701 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:12:41.702 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:41.702 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:41.702 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:41.702 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:41.702 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:41.703 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:41.703 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:12:41.703 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:41.703 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:41.704 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:41.704 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:12:41.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:41.704 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:12:41.704 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:12:41.705 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:12:41.705 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:41.705 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:12:41.706 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:12:41.707 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:12:41.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:41.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:41.709 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:41.753 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:41.754 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:41.816 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:41.816 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:41.816 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:41.816 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:41.817 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:41.817 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:41.817 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:41.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:41.818 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:41.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:41.818 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:41.818 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:41.819 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:41.819 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910356.178033_633.1", 2022-01-11T14:12:41.819 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:41.819 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:12:41.819 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:41.820 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:36.178033Z", 2022-01-11T14:12:41.820 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:41.820 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:41.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:41.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:41.821 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:41.821 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:41.821 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:41.821 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:41.822 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:41.822 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:41.822 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:41.822 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:41.823 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:41.823 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:41.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:41.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:41.824 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:41.824 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:41.824 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:41.824 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:41.825 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:41.825 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:41.825 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:41.825 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:41.825 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:41.826 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:41.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:41.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:41.826 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:41.827 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:41.827 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:41.827 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:41.827 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:41.827 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:41.828 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:41.828 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:41.828 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:41.828 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:41.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:41.829 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:41.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:41.829 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:41.829 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:12:41.830 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:12:41.830 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:41.830 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:41.832 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:12:41.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:41.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:41.880 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:41.931 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:41.932 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:41.995 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:41.996+0000 7f530aeaf500 1 waiting for incremental sync to catch up: 2022-01-11T14:12:41.995 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2022-01-11T14:12:41.995 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000005.19.17, , ] 2022-01-11T14:12:45.353 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:12:45.355+0000 7fc1e3ebe700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2022-01-11T14:12:46.996 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:46.998+0000 7f530aeaf500 1 bucket sync caught up with source: 2022-01-11T14:12:46.996 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000005.19.17, , ] 2022-01-11T14:12:46.997 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000005.19.17, , ] 2022-01-11T14:12:46.997 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:46.998+0000 7f530aeaf500 0 bucket checkpoint complete 2022-01-11T14:12:47.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:47.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:47.016 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:47.072 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:47.072 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:47.138 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:47.140+0000 7f91f4ad3500 1 bucket sync caught up with source: 2022-01-11T14:12:47.138 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000005.19.17, , ] 2022-01-11T14:12:47.139 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000005.19.17, , ] 2022-01-11T14:12:47.139 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:47.140+0000 7f91f4ad3500 0 bucket checkpoint complete 2022-01-11T14:12:47.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:47.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:47.154 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:47.210 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:47.210 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:47.262 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:47.263+0000 7f502346e500 1 bucket sync caught up with source: 2022-01-11T14:12:47.262 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000005.19.17, , ] 2022-01-11T14:12:47.263 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000005.19.17, , ] 2022-01-11T14:12:47.263 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:47.263+0000 7f502346e500 0 bucket checkpoint complete 2022-01-11T14:12:47.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:47.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:47.280 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:47.330 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:47.330 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:47.376 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:47.378+0000 7f20777af500 1 bucket sync caught up with source: 2022-01-11T14:12:47.377 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000007.318.14, , ] 2022-01-11T14:12:47.377 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000007.318.14, , ] 2022-01-11T14:12:47.377 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:47.378+0000 7f20777af500 0 bucket checkpoint complete 2022-01-11T14:12:47.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:47.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:47.390 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:47.441 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:47.441 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:47.486 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:47.488+0000 7f2acf20d500 1 bucket sync caught up with source: 2022-01-11T14:12:47.486 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000007.318.14, , ] 2022-01-11T14:12:47.486 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000007.318.14, , ] 2022-01-11T14:12:47.487 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:47.488+0000 7f2acf20d500 0 bucket checkpoint complete 2022-01-11T14:12:47.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:47.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:47.501 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:47.550 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:47.551 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:47.612 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:47.614+0000 7f03eeee7500 1 bucket sync caught up with source: 2022-01-11T14:12:47.612 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000007.286.14, , ] 2022-01-11T14:12:47.613 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000007.286.14, , ] 2022-01-11T14:12:47.613 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:47.614+0000 7f03eeee7500 0 bucket checkpoint complete 2022-01-11T14:12:47.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:47.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:47.635 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:47.687 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:47.687 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:47.735 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:47.738+0000 7f830b82f500 1 bucket sync caught up with source: 2022-01-11T14:12:47.735 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000007.286.14, , ] 2022-01-11T14:12:47.736 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000007.286.14, , ] 2022-01-11T14:12:47.736 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:47.738+0000 7f830b82f500 0 bucket checkpoint complete 2022-01-11T14:12:47.748 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-20 zones={test-zone2, test-zone1} 2022-01-11T14:12:47.754 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:47.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:47.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:47.800 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:47.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:47.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:47.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:47.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:47.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:47.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:47.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:48.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:48.103 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-20 zones={test-zone2, test-zone1} 2022-01-11T14:12:48.104 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-20 zones={test-zone3, test-zone1} 2022-01-11T14:12:48.110 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:48.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:48.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:48.156 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:48.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:48.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:48.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:48.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:48.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:48.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:48.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:48.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:48.462 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-20 zones={test-zone3, test-zone1} 2022-01-11T14:12:48.463 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-20 zones={test-zone3, test-zone2} 2022-01-11T14:12:48.469 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:48.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:48.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:48.521 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:48.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:48.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:48.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:48.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:48.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:48.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:48.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:48.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:48.831 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-20 zones={test-zone3, test-zone2} 2022-01-11T14:12:48.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2022-01-11T14:12:48.832 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-21 2022-01-11T14:12:48.847 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-22 2022-01-11T14:12:48.884 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-23 2022-01-11T14:12:48.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:48.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:48.971 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:49.014 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:49.015 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:49.065 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:12:49.085 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:12:49.086 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:49.086 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910368.963730_651.1", 2022-01-11T14:12:49.087 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:12:48.963730Z" 2022-01-11T14:12:49.087 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:49.088 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:49.088 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:49.088 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:12:49.089 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:49.089 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:49.090 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:49.090 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:49.091 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:49.091 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:49.092 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:49.092 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:49.092 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:49.093 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:12:49.094 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:12:49.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:49.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:49.097 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:49.141 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:49.142 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:49.204 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:49.205 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:49.205 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:49.205 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:49.205 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:49.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:49.206 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:49.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:49.206 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:49.206 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910368.843035_634.1", 2022-01-11T14:12:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:12:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:48.843035Z", 2022-01-11T14:12:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:49.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:49.215 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:49.215 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:49.215 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:12:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:12:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:49.219 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:49.220 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910368.963730_651.1 target=1_1641910368.843035_634.1 2022-01-11T14:12:54.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:54.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:54.223 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:54.273 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:54.273 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:54.343 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:54.343 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:54.343 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:54.344 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:54.344 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:54.344 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:54.344 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:54.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:54.346 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:54.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:54.347 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:54.347 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:54.347 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:54.347 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910368.963730_651.1", 2022-01-11T14:12:54.347 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:54.348 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:12:54.348 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:54.348 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:48.963730Z", 2022-01-11T14:12:54.348 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:54.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:54.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:54.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:54.350 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:54.350 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:54.351 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:54.351 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:54.352 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:54.352 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:54.353 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:54.353 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:54.354 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:54.354 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:54.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:54.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:54.355 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:54.356 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:54.356 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:54.357 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:54.357 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:54.357 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:54.358 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:54.358 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:54.359 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:54.359 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:54.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:54.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:54.360 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:54.361 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:54.361 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:54.361 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:54.362 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:54.362 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:54.362 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:54.362 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:54.362 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:54.363 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:54.363 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:54.363 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:54.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:54.364 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:54.364 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:12:54.364 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:12:54.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:54.364 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:54.366 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:12:54.366 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:12:54.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:54.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:54.368 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:54.416 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:54.416 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:54.474 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:12:54.474 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:12:54.475 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:12:54.475 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:12:54.475 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:12:54.475 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:54.476 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:54.476 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:54.476 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:12:54.476 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:54.477 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:12:54.477 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:54.477 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:54.477 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910368.963730_651.1", 2022-01-11T14:12:54.477 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:54.478 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:12:54.478 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:54.478 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:12:48.963730Z", 2022-01-11T14:12:54.478 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:54.479 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:54.479 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:54.479 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:54.479 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:12:54.480 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:54.480 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:54.480 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:54.480 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:54.481 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:54.481 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:54.481 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:54.481 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:54.482 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:54.482 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:54.482 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:54.482 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:12:54.482 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:54.483 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:54.483 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:54.483 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:54.483 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:54.484 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:54.484 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:12:54.484 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:54.484 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:54.485 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:54.485 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:54.485 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:12:54.485 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:12:54.485 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:12:54.486 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:54.486 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:12:54.486 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:12:54.486 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:12:54.487 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:12:54.487 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:12:54.487 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:54.487 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:54.487 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:12:54.488 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:54.488 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:12:54.488 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:12:54.488 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:12:54.489 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:54.489 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:12:54.490 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:12:54.491 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:12:54.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:54.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:54.492 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:54.537 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:54.537 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:54.601 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:54.603 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:54.604 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:54.604 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:54.604 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:54.604 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:54.605 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:54.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:54.605 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:54.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:54.606 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:54.606 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:54.606 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:54.607 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910368.963730_651.1", 2022-01-11T14:12:54.607 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:54.607 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:12:54.607 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:54.608 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:48.963730Z", 2022-01-11T14:12:54.608 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:54.608 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:54.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:54.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:54.609 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:54.609 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:54.609 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:54.609 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:54.610 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:54.610 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:54.610 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:54.610 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:54.610 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:54.611 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:54.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:54.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:54.611 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:54.612 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:54.612 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:54.612 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:54.612 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:54.612 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:54.613 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:54.613 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:54.613 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:54.613 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:54.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:54.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:54.614 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:54.614 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:54.614 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:54.615 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:54.615 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:54.615 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:54.615 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:54.616 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:54.616 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:54.616 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:54.616 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:54.616 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:54.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:54.617 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:54.617 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:12:54.617 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:12:54.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:54.618 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:54.619 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:12:54.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:54.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:54.806 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:54.857 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:54.857 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:54.926 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:54.929+0000 7fb84e8b0500 1 bucket sync caught up with source: 2022-01-11T14:12:54.926 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.13.11, , ] 2022-01-11T14:12:54.927 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.13.11, , ] 2022-01-11T14:12:54.927 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:54.929+0000 7fb84e8b0500 0 bucket checkpoint complete 2022-01-11T14:12:54.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:54.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:54.942 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:55.006 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:55.006 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:55.093 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:55.095+0000 7f267305a500 1 bucket sync caught up with source: 2022-01-11T14:12:55.093 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000003.13.11, , ] 2022-01-11T14:12:55.093 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000003.13.11, , ] 2022-01-11T14:12:55.094 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:55.095+0000 7f267305a500 0 bucket checkpoint complete 2022-01-11T14:12:55.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:55.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:55.109 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:55.160 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:55.160 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:55.221 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:55.224+0000 7f59126ba500 1 bucket sync caught up with source: 2022-01-11T14:12:55.221 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.13.11, , ] 2022-01-11T14:12:55.222 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.13.11, , ] 2022-01-11T14:12:55.222 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:55.224+0000 7f59126ba500 0 bucket checkpoint complete 2022-01-11T14:12:55.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:55.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:55.236 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:55.293 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:55.294 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:55.353 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:55.356+0000 7f2ed3317500 1 bucket sync caught up with source: 2022-01-11T14:12:55.354 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000002.244.5, , ] 2022-01-11T14:12:55.354 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000002.244.5, , ] 2022-01-11T14:12:55.354 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:55.356+0000 7f2ed3317500 0 bucket checkpoint complete 2022-01-11T14:12:55.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:55.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:55.372 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:55.426 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:55.426 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:55.493 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:55.495+0000 7f676dd78500 1 bucket sync caught up with source: 2022-01-11T14:12:55.493 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000002.244.5, , ] 2022-01-11T14:12:55.494 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000002.244.5, , ] 2022-01-11T14:12:55.494 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:55.495+0000 7f676dd78500 0 bucket checkpoint complete 2022-01-11T14:12:55.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:55.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:55.519 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:55.568 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:55.568 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:55.629 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:55.631+0000 7fa3ba6c4500 1 bucket sync caught up with source: 2022-01-11T14:12:55.630 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000002.215.5, , ] 2022-01-11T14:12:55.630 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000002.215.5, , ] 2022-01-11T14:12:55.630 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:55.631+0000 7fa3ba6c4500 0 bucket checkpoint complete 2022-01-11T14:12:55.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:55.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:55.644 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:55.693 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:55.694 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:55.764 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:55.767+0000 7f67b9b42500 1 bucket sync caught up with source: 2022-01-11T14:12:55.765 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000002.215.5, , ] 2022-01-11T14:12:55.765 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000002.215.5, , ] 2022-01-11T14:12:55.765 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:55.767+0000 7f67b9b42500 0 bucket checkpoint complete 2022-01-11T14:12:55.784 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-21 zones={test-zone2, test-zone1} 2022-01-11T14:12:55.792 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:55.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:55.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:55.844 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:55.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:55.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:55.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:55.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:55.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:55.976 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-21 zones={test-zone2, test-zone1} 2022-01-11T14:12:55.976 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-21 zones={test-zone3, test-zone1} 2022-01-11T14:12:55.983 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:56.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:56.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:56.032 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:56.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:56.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:56.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:56.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:56.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:56.164 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-21 zones={test-zone3, test-zone1} 2022-01-11T14:12:56.164 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-21 zones={test-zone3, test-zone2} 2022-01-11T14:12:56.171 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:56.219 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:56.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:56.220 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:56.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:56.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:56.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:56.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:56.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:56.356 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-21 zones={test-zone3, test-zone2} 2022-01-11T14:12:56.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:56.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:56.358 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:56.408 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:56.408 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:56.465 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:56.468+0000 7f45427d5500 1 bucket sync caught up with source: 2022-01-11T14:12:56.466 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000002.10.5, , ] 2022-01-11T14:12:56.466 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000002.10.5, , ] 2022-01-11T14:12:56.466 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:56.468+0000 7f45427d5500 0 bucket checkpoint complete 2022-01-11T14:12:56.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:56.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:56.486 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:56.535 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:56.536 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:56.604 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:56.606+0000 7f823936a500 1 bucket sync caught up with source: 2022-01-11T14:12:56.604 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000002.10.5, , ] 2022-01-11T14:12:56.604 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000002.10.5, , ] 2022-01-11T14:12:56.604 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:56.606+0000 7f823936a500 0 bucket checkpoint complete 2022-01-11T14:12:56.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:56.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:56.622 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:56.671 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:56.672 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:56.729 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:56.732+0000 7f733949c500 1 bucket sync caught up with source: 2022-01-11T14:12:56.729 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000002.10.5, , ] 2022-01-11T14:12:56.730 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000002.10.5, , ] 2022-01-11T14:12:56.730 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:56.732+0000 7f733949c500 0 bucket checkpoint complete 2022-01-11T14:12:56.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:56.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:56.751 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:56.801 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:56.802 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:56.866 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:56.868+0000 7f1852a85500 1 bucket sync caught up with source: 2022-01-11T14:12:56.867 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000003.168.11, , ] 2022-01-11T14:12:56.867 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000003.168.11, , ] 2022-01-11T14:12:56.867 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:56.868+0000 7f1852a85500 0 bucket checkpoint complete 2022-01-11T14:12:56.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:56.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:56.880 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:56.930 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:56.931 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:56.976 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:56.978+0000 7f20f7532500 1 bucket sync caught up with source: 2022-01-11T14:12:56.977 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000003.168.11, , ] 2022-01-11T14:12:56.977 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000003.168.11, , ] 2022-01-11T14:12:56.977 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:56.978+0000 7f20f7532500 0 bucket checkpoint complete 2022-01-11T14:12:56.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:56.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:56.990 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:57.040 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:57.040 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:57.100 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:57.102+0000 7f959d7f8500 1 bucket sync caught up with source: 2022-01-11T14:12:57.100 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000002.124.5, , ] 2022-01-11T14:12:57.101 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000002.124.5, , ] 2022-01-11T14:12:57.101 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:57.102+0000 7f959d7f8500 0 bucket checkpoint complete 2022-01-11T14:12:57.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:57.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:57.123 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:57.173 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:57.173 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:57.246 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:57.249+0000 7fbbe6c1d500 1 bucket sync caught up with source: 2022-01-11T14:12:57.247 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000002.124.5, , ] 2022-01-11T14:12:57.247 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000002.124.5, , ] 2022-01-11T14:12:57.247 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:57.249+0000 7fbbe6c1d500 0 bucket checkpoint complete 2022-01-11T14:12:57.264 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-22 zones={test-zone2, test-zone1} 2022-01-11T14:12:57.271 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:57.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:57.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:57.320 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:57.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:57.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:57.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:57.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:57.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:57.452 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-22 zones={test-zone2, test-zone1} 2022-01-11T14:12:57.453 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-22 zones={test-zone3, test-zone1} 2022-01-11T14:12:57.460 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:57.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:57.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:57.508 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:57.512 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:57.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:57.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:57.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:57.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:57.643 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-22 zones={test-zone3, test-zone1} 2022-01-11T14:12:57.644 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-22 zones={test-zone3, test-zone2} 2022-01-11T14:12:57.650 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:57.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:57.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:57.696 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:57.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:57.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:57.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:57.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:57.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:57.835 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-22 zones={test-zone3, test-zone2} 2022-01-11T14:12:57.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:57.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:57.837 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:57.888 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:57.888 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:57.946 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:57.949+0000 7f9ee9b6f500 1 bucket sync caught up with source: 2022-01-11T14:12:57.947 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000002.12.5, , ] 2022-01-11T14:12:57.947 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000002.12.5, , ] 2022-01-11T14:12:57.947 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:57.949+0000 7f9ee9b6f500 0 bucket checkpoint complete 2022-01-11T14:12:57.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:57.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:57.970 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:58.019 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:58.020 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:58.066 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:58.069+0000 7f6bf6948500 1 bucket sync caught up with source: 2022-01-11T14:12:58.067 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000002.12.5, , ] 2022-01-11T14:12:58.067 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000002.12.5, , ] 2022-01-11T14:12:58.067 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:58.069+0000 7f6bf6948500 0 bucket checkpoint complete 2022-01-11T14:12:58.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:58.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:58.080 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:58.132 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:58.132 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:58.186 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:58.189+0000 7f1121d60500 1 bucket sync caught up with source: 2022-01-11T14:12:58.186 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000002.12.5, , ] 2022-01-11T14:12:58.186 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000002.12.5, , ] 2022-01-11T14:12:58.187 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:58.189+0000 7f1121d60500 0 bucket checkpoint complete 2022-01-11T14:12:58.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:58.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:58.209 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:58.260 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:58.260 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:58.309 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:58.311+0000 7f7b20036500 1 bucket sync caught up with source: 2022-01-11T14:12:58.310 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000002.349.5, , ] 2022-01-11T14:12:58.310 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000002.349.5, , ] 2022-01-11T14:12:58.310 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:58.311+0000 7f7b20036500 0 bucket checkpoint complete 2022-01-11T14:12:58.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:58.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:58.323 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:58.373 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:58.373 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:58.425 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:58.427+0000 7fe0613cb500 1 bucket sync caught up with source: 2022-01-11T14:12:58.425 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000002.349.5, , ] 2022-01-11T14:12:58.425 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000002.349.5, , ] 2022-01-11T14:12:58.426 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:58.427+0000 7fe0613cb500 0 bucket checkpoint complete 2022-01-11T14:12:58.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:58.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:58.450 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:58.498 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:58.499 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:58.560 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:58.562+0000 7f8c4b5f7500 1 bucket sync caught up with source: 2022-01-11T14:12:58.560 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000003.425.11, , ] 2022-01-11T14:12:58.560 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000003.425.11, , ] 2022-01-11T14:12:58.560 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:12:58.562+0000 7f8c4b5f7500 0 bucket checkpoint complete 2022-01-11T14:12:58.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:58.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:12:58.575 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:12:58.624 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:58.625 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:58.687 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:58.690+0000 7f4b47b13500 1 bucket sync caught up with source: 2022-01-11T14:12:58.687 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.425.11, , ] 2022-01-11T14:12:58.688 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.425.11, , ] 2022-01-11T14:12:58.688 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:12:58.690+0000 7f4b47b13500 0 bucket checkpoint complete 2022-01-11T14:12:58.700 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-23 zones={test-zone2, test-zone1} 2022-01-11T14:12:58.708 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:58.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:58.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:58.756 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:58.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:58.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:58.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:58.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:58.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:58.888 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-23 zones={test-zone2, test-zone1} 2022-01-11T14:12:58.889 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-23 zones={test-zone3, test-zone1} 2022-01-11T14:12:58.895 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:58.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:58.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:58.944 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:58.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:58.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:58.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:59.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:59.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:59.076 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-23 zones={test-zone3, test-zone1} 2022-01-11T14:12:59.077 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-23 zones={test-zone3, test-zone2} 2022-01-11T14:12:59.083 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:12:59.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:59.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:59.132 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:12:59.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:59.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:12:59.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:59.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:59.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:12:59.263 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-23 zones={test-zone3, test-zone2} 2022-01-11T14:12:59.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2022-01-11T14:12:59.265 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-24 2022-01-11T14:12:59.288 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-25 2022-01-11T14:12:59.327 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-26 2022-01-11T14:12:59.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:59.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:59.426 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:59.468 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:59.469 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:59.519 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:12:59.533 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:12:59.534 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:59.534 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910379.425023_675.1", 2022-01-11T14:12:59.534 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:12:59.425023Z" 2022-01-11T14:12:59.534 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:59.535 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:59.535 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:59.535 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:12:59.539 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:59.540 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:59.540 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:59.540 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:59.540 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:12:59.541 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:12:59.541 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:12:59.542 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:12:59.542 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:12:59.542 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:12:59.543 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:12:59.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:59.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:12:59.545 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:12:59.589 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:12:59.589 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:12:59.658 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:12:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:12:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:12:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:12:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:12:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:12:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:12:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:12:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:59.662 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910379.275386_652.1", 2022-01-11T14:12:59.662 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:59.662 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:12:59.662 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:59.663 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:59.275386Z", 2022-01-11T14:12:59.663 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:59.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:59.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:59.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:59.664 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:12:59.664 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:59.664 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:59.664 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:12:59.665 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:59.665 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:59.665 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:59.665 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:12:59.665 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:59.666 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:59.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:59.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:59.666 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:12:59.667 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:59.667 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:59.667 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:59.667 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:59.668 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:59.668 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:59.668 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:59.668 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:59.669 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:59.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:59.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:12:59.669 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:12:59.669 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:12:59.670 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:12:59.670 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:12:59.670 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:12:59.670 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:12:59.671 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:12:59.671 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:12:59.671 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:12:59.671 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:59.672 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:59.672 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:12:59.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:12:59.672 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:12:59.672 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:12:59.673 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:12:59.673 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:12:59.673 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:12:59.674 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910379.425023_675.1 target=1_1641910379.275386_652.1 2022-01-11T14:13:04.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:04.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:04.677 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:13:04.726 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:04.726 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:04.799 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:13:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:13:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:13:04.800 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:13:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:13:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:13:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:04.801 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:13:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:13:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:04.802 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910379.425023_675.1", 2022-01-11T14:13:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:13:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:04.803 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:59.425023Z", 2022-01-11T14:13:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:04.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:13:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:13:04.805 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:13:04.806 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:13:04.807 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:04.808 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:04.808 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:04.808 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:04.808 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:04.808 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:04.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:13:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:04.810 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:04.811 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:13:04.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:13:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:13:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:13:04.813 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:04.813 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:13:04.815 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2022-01-11T14:13:04.815 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2022-01-11T14:13:04.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:04.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:04.817 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:13:04.864 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:04.864 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:04.924 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:13:04.924 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": { 2022-01-11T14:13:04.925 INFO:teuthology.orchestra.run.smithi161.stdout: "info": { 2022-01-11T14:13:04.925 INFO:teuthology.orchestra.run.smithi161.stdout: "status": "sync", 2022-01-11T14:13:04.925 INFO:teuthology.orchestra.run.smithi161.stdout: "num_shards": 4, 2022-01-11T14:13:04.925 INFO:teuthology.orchestra.run.smithi161.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:13:04.926 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:13:04.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:13:04.926 INFO:teuthology.orchestra.run.smithi161.stdout: "markers": [ 2022-01-11T14:13:04.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:13:04.927 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 0, 2022-01-11T14:13:04.927 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:13:04.927 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:13:04.927 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910379.425023_675.1", 2022-01-11T14:13:04.928 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:13:04.928 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 1, 2022-01-11T14:13:04.928 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:13:04.928 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:12:59.425023Z", 2022-01-11T14:13:04.928 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:13:04.929 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:13:04.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:13:04.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:13:04.930 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 1, 2022-01-11T14:13:04.930 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:13:04.930 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:13:04.930 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:13:04.931 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:13:04.931 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:13:04.931 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:13:04.931 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:13:04.932 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:13:04.932 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:13:04.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:13:04.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:13:04.933 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 2, 2022-01-11T14:13:04.933 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:13:04.933 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:13:04.933 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:13:04.934 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:13:04.934 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:13:04.934 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:13:04.934 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:13:04.934 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:13:04.935 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:13:04.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:13:04.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:13:04.935 INFO:teuthology.orchestra.run.smithi161.stdout: "key": 3, 2022-01-11T14:13:04.936 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:13:04.936 INFO:teuthology.orchestra.run.smithi161.stdout: "state": 1, 2022-01-11T14:13:04.936 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:13:04.936 INFO:teuthology.orchestra.run.smithi161.stdout: "next_step_marker": "", 2022-01-11T14:13:04.937 INFO:teuthology.orchestra.run.smithi161.stdout: "total_entries": 0, 2022-01-11T14:13:04.937 INFO:teuthology.orchestra.run.smithi161.stdout: "pos": 0, 2022-01-11T14:13:04.937 INFO:teuthology.orchestra.run.smithi161.stdout: "timestamp": "0.000000", 2022-01-11T14:13:04.937 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:13:04.938 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:13:04.938 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:13:04.938 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:13:04.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:13:04.939 INFO:teuthology.orchestra.run.smithi161.stdout: "full_sync": { 2022-01-11T14:13:04.939 INFO:teuthology.orchestra.run.smithi161.stdout: "total": 1, 2022-01-11T14:13:04.939 INFO:teuthology.orchestra.run.smithi161.stdout: "complete": 1 2022-01-11T14:13:04.939 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:13:04.939 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:13:04.941 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2022-01-11T14:13:04.941 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2022-01-11T14:13:04.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:04.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:04.943 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:13:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:04.987 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:05.056 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:13:05.056 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:13:05.057 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:13:05.057 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:13:05.057 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:13:05.057 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:13:05.058 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:05.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:05.058 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:13:05.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:05.059 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:13:05.059 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:05.059 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:05.059 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910379.425023_675.1", 2022-01-11T14:13:05.060 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:05.060 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:13:05.060 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:05.060 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:12:59.425023Z", 2022-01-11T14:13:05.060 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:05.061 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:05.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:05.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:05.061 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:13:05.062 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:05.062 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:05.062 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:13:05.062 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:05.062 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:05.063 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:05.063 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:13:05.063 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:05.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:05.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:05.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:05.064 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:13:05.064 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:05.065 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:05.065 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:05.065 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:05.065 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:05.065 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:05.066 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:05.066 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:05.066 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:05.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:05.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:05.067 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:13:05.067 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:05.067 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:05.067 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:05.068 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:05.068 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:05.068 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:05.068 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:05.069 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:05.069 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:05.069 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:05.069 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:13:05.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:05.070 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:13:05.070 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:13:05.070 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:13:05.071 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:05.071 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:13:05.072 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2022-01-11T14:13:05.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:05.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:05.278 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:05.330 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:05.331 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:05.387 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:05.389+0000 7f8a47167500 1 bucket sync caught up with source: 2022-01-11T14:13:05.388 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.29.6, , ] 2022-01-11T14:13:05.388 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.29.6, , ] 2022-01-11T14:13:05.388 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:05.389+0000 7f8a47167500 0 bucket checkpoint complete 2022-01-11T14:13:05.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:05.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:05.418 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:05.475 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:05.477 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:05.536 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:05.538+0000 7fcd2f821500 1 bucket sync caught up with source: 2022-01-11T14:13:05.536 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000003.29.6, , ] 2022-01-11T14:13:05.537 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000003.29.6, , ] 2022-01-11T14:13:05.537 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:05.538+0000 7fcd2f821500 0 bucket checkpoint complete 2022-01-11T14:13:05.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:05.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:05.556 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:05.607 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:05.608 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:05.657 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:05.659+0000 7f1308496500 1 bucket sync caught up with source: 2022-01-11T14:13:05.658 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.29.6, , ] 2022-01-11T14:13:05.658 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.29.6, , ] 2022-01-11T14:13:05.658 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:05.659+0000 7f1308496500 0 bucket checkpoint complete 2022-01-11T14:13:05.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:05.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:05.671 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:05.723 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:05.723 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:05.774 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:05.776+0000 7fe5e86c0500 1 bucket sync caught up with source: 2022-01-11T14:13:05.774 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000001.411.12, , ] 2022-01-11T14:13:05.774 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000001.411.12, , ] 2022-01-11T14:13:05.775 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:05.776+0000 7fe5e86c0500 0 bucket checkpoint complete 2022-01-11T14:13:05.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:05.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:05.792 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:05.841 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:05.841 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:05.903 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:05.905+0000 7f5cd90c3500 1 bucket sync caught up with source: 2022-01-11T14:13:05.903 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000001.411.12, , ] 2022-01-11T14:13:05.903 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000001.411.12, , ] 2022-01-11T14:13:05.904 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:05.905+0000 7f5cd90c3500 0 bucket checkpoint complete 2022-01-11T14:13:05.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:05.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:05.917 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:05.966 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:05.967 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:06.019 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:06.021+0000 7f0764513500 1 bucket sync caught up with source: 2022-01-11T14:13:06.020 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000001.352.12, , ] 2022-01-11T14:13:06.020 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000001.352.12, , ] 2022-01-11T14:13:06.020 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:06.021+0000 7f0764513500 0 bucket checkpoint complete 2022-01-11T14:13:06.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:06.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:06.045 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:06.094 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:06.095 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:06.148 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:06.150+0000 7f660eb87500 1 bucket sync caught up with source: 2022-01-11T14:13:06.148 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.352.12, , ] 2022-01-11T14:13:06.148 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.352.12, , ] 2022-01-11T14:13:06.149 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:06.150+0000 7f660eb87500 0 bucket checkpoint complete 2022-01-11T14:13:06.166 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-24 zones={test-zone2, test-zone1} 2022-01-11T14:13:06.173 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:13:06.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:06.224 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:13:06.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:06.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:13:06.279 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-24 zones={test-zone2, test-zone1} 2022-01-11T14:13:06.280 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-24 zones={test-zone3, test-zone1} 2022-01-11T14:13:06.286 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:13:06.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:06.331 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:13:06.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:06.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:13:06.392 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-24 zones={test-zone3, test-zone1} 2022-01-11T14:13:06.392 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-24 zones={test-zone3, test-zone2} 2022-01-11T14:13:06.398 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:13:06.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:06.443 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:13:06.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:06.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:13:06.503 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-24 zones={test-zone3, test-zone2} 2022-01-11T14:13:06.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:06.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:06.505 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:06.554 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:06.555 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:06.601 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:06.603+0000 7f2baaf21500 1 bucket sync caught up with source: 2022-01-11T14:13:06.602 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.13.12, , ] 2022-01-11T14:13:06.602 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.13.12, , ] 2022-01-11T14:13:06.602 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:06.603+0000 7f2baaf21500 0 bucket checkpoint complete 2022-01-11T14:13:06.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:06.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:06.619 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:06.668 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:06.669 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:06.732 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:06.734+0000 7f5d4883b500 1 bucket sync caught up with source: 2022-01-11T14:13:06.732 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000001.13.12, , ] 2022-01-11T14:13:06.733 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000001.13.12, , ] 2022-01-11T14:13:06.733 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:06.734+0000 7f5d4883b500 0 bucket checkpoint complete 2022-01-11T14:13:06.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:06.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:06.760 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:06.810 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:06.810 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:06.864 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:06.866+0000 7f5015aa4500 1 bucket sync caught up with source: 2022-01-11T14:13:06.865 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.13.12, , ] 2022-01-11T14:13:06.865 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.13.12, , ] 2022-01-11T14:13:06.865 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:06.866+0000 7f5015aa4500 0 bucket checkpoint complete 2022-01-11T14:13:06.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:06.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:06.886 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:06.936 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:06.936 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:06.994 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:06.996+0000 7f77b1965500 1 bucket sync caught up with source: 2022-01-11T14:13:06.994 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000003.381.6, , ] 2022-01-11T14:13:06.995 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000003.381.6, , ] 2022-01-11T14:13:06.995 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:06.996+0000 7f77b1965500 0 bucket checkpoint complete 2022-01-11T14:13:07.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:07.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:07.009 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:07.058 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:07.058 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:07.106 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:07.108+0000 7fedfce7f500 1 bucket sync caught up with source: 2022-01-11T14:13:07.106 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000003.381.6, , ] 2022-01-11T14:13:07.107 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000003.381.6, , ] 2022-01-11T14:13:07.107 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:07.108+0000 7fedfce7f500 0 bucket checkpoint complete 2022-01-11T14:13:07.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:07.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:07.123 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:07.171 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:07.172 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:07.236 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:07.238+0000 7f58a6563500 1 bucket sync caught up with source: 2022-01-11T14:13:07.236 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000001.347.12, , ] 2022-01-11T14:13:07.236 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000001.347.12, , ] 2022-01-11T14:13:07.236 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:07.238+0000 7f58a6563500 0 bucket checkpoint complete 2022-01-11T14:13:07.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:07.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:07.258 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:07.308 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:07.309 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:07.365 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:07.367+0000 7f40d354e500 1 bucket sync caught up with source: 2022-01-11T14:13:07.365 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.347.12, , ] 2022-01-11T14:13:07.365 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.347.12, , ] 2022-01-11T14:13:07.365 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:07.367+0000 7f40d354e500 0 bucket checkpoint complete 2022-01-11T14:13:07.378 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-25 zones={test-zone2, test-zone1} 2022-01-11T14:13:07.384 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:13:07.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:07.432 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:13:07.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:07.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:13:07.492 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-25 zones={test-zone2, test-zone1} 2022-01-11T14:13:07.494 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-25 zones={test-zone3, test-zone1} 2022-01-11T14:13:07.500 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:13:07.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:07.548 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:13:07.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:07.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:13:07.614 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-25 zones={test-zone3, test-zone1} 2022-01-11T14:13:07.615 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-25 zones={test-zone3, test-zone2} 2022-01-11T14:13:07.621 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:13:07.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:07.668 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:13:07.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:07.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:13:07.728 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-25 zones={test-zone3, test-zone2} 2022-01-11T14:13:07.728 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:07.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:07.730 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:07.781 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:07.782 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:07.835 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:07.837+0000 7fac1978b500 1 bucket sync caught up with source: 2022-01-11T14:13:07.835 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.27.12, , ] 2022-01-11T14:13:07.835 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.27.12, , ] 2022-01-11T14:13:07.835 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:07.837+0000 7fac1978b500 0 bucket checkpoint complete 2022-01-11T14:13:07.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:07.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:07.856 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:07.907 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:07.908 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:07.956 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:07.958+0000 7fde28e26500 1 bucket sync caught up with source: 2022-01-11T14:13:07.957 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000001.27.12, , ] 2022-01-11T14:13:07.957 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000001.27.12, , ] 2022-01-11T14:13:07.957 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:07.958+0000 7fde28e26500 0 bucket checkpoint complete 2022-01-11T14:13:07.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:07.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:07.971 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:08.022 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:08.022 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:08.084 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:08.086+0000 7f7a8b0a6500 1 bucket sync caught up with source: 2022-01-11T14:13:08.084 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.27.12, , ] 2022-01-11T14:13:08.084 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.27.12, , ] 2022-01-11T14:13:08.085 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:08.086+0000 7f7a8b0a6500 0 bucket checkpoint complete 2022-01-11T14:13:08.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:08.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:08.105 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:08.154 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:08.155 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:08.214 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:08.215+0000 7f3309d35500 1 bucket sync caught up with source: 2022-01-11T14:13:08.214 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000001.187.12, , ] 2022-01-11T14:13:08.214 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000001.187.12, , ] 2022-01-11T14:13:08.215 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:08.215+0000 7f3309d35500 0 bucket checkpoint complete 2022-01-11T14:13:08.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:08.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:08.228 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:08.278 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:08.278 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:08.325 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:08.327+0000 7efe576ce500 1 bucket sync caught up with source: 2022-01-11T14:13:08.326 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000001.187.12, , ] 2022-01-11T14:13:08.326 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000001.187.12, , ] 2022-01-11T14:13:08.326 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:08.327+0000 7efe576ce500 0 bucket checkpoint complete 2022-01-11T14:13:08.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:08.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:08.344 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket epmlcl-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:08.394 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:08.394 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:08.456 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:08.458+0000 7f0e95645500 1 bucket sync caught up with source: 2022-01-11T14:13:08.457 INFO:teuthology.orchestra.run.smithi161.stderr: local status: [, , , , , , , , 00000000003.373.6, , ] 2022-01-11T14:13:08.457 INFO:teuthology.orchestra.run.smithi161.stderr: remote markers: [, , , , , , , , 00000000003.373.6, , ] 2022-01-11T14:13:08.457 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:13:08.458+0000 7f0e95645500 0 bucket checkpoint complete 2022-01-11T14:13:08.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:08.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'epmlcl-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2022-01-11T14:13:08.473 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket epmlcl-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2022-01-11T14:13:08.523 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:08.524 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:08.579 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:08.581+0000 7f1fde7aa500 1 bucket sync caught up with source: 2022-01-11T14:13:08.579 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.373.6, , ] 2022-01-11T14:13:08.580 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.373.6, , ] 2022-01-11T14:13:08.580 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:13:08.581+0000 7f1fde7aa500 0 bucket checkpoint complete 2022-01-11T14:13:08.592 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-26 zones={test-zone2, test-zone1} 2022-01-11T14:13:08.599 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:13:08.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:08.644 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:13:08.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:08.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:13:08.703 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-26 zones={test-zone2, test-zone1} 2022-01-11T14:13:08.704 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-26 zones={test-zone3, test-zone1} 2022-01-11T14:13:08.710 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:13:08.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:08.756 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:13:08.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:08.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:13:08.815 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-26 zones={test-zone3, test-zone1} 2022-01-11T14:13:08.816 INFO:tasks.rgw_multi.zone_rados:comparing bucket=epmlcl-26 zones={test-zone3, test-zone2} 2022-01-11T14:13:08.822 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2022-01-11T14:13:08.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:08.868 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2022-01-11T14:13:08.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2022-01-11T14:13:08.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2022-01-11T14:13:08.929 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=epmlcl-26 zones={test-zone3, test-zone2} 2022-01-11T14:13:08.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2022-01-11T14:13:08.930 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-27 2022-01-11T14:13:08.950 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-28 2022-01-11T14:13:08.986 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-29 2022-01-11T14:13:09.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2022-01-11T14:13:09.188 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-30 2022-01-11T14:13:09.210 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-31 2022-01-11T14:13:09.241 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-32 2022-01-11T14:13:09.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2022-01-11T14:13:09.464 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-33 2022-01-11T14:13:09.495 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-34 2022-01-11T14:13:09.552 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-35 2022-01-11T14:13:11.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2022-01-11T14:13:11.856 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-36 2022-01-11T14:13:11.900 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-37 2022-01-11T14:13:11.965 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-38 2022-01-11T14:13:12.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:12.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:12.004 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:13:12.047 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:12.048 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:12.090 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:13:12.103 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:13:12.104 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:13:12.104 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910391.991941_746.1", 2022-01-11T14:13:12.104 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:13:11.991941Z" 2022-01-11T14:13:12.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:13:12.105 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:13:12.105 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:13:12.105 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:13:12.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:13:12.106 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:13:12.106 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:13:12.106 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:13:12.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:13:12.107 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:13:12.107 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:13:12.107 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:13:12.107 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:13:12.108 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:13:12.109 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:13:12.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:12.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:12.111 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:13:12.154 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:12.154 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:12.214 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:13:12.214 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:13:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:13:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:13:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:13:12.215 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:13:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:13:12.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:13:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:12.217 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.204054_696.1", 2022-01-11T14:13:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:13:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:12.218 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.204054Z", 2022-01-11T14:13:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:12.219 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:13:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:13:12.220 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:13:12.221 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:12.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:13:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:12.223 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:12.224 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:12.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:13:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:12.226 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:12.227 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:13:12.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:13:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:13:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:13:12.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:12.230 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:13:12.231 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.204054_696.1 2022-01-11T14:13:17.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:17.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:17.234 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:13:17.277 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:17.278 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:17.395 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:13:17.396 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:13:17.396 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:13:17.396 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:13:17.396 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:13:17.397 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:13:17.397 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:17.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:17.397 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:13:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:13:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.949237_678.1", 2022-01-11T14:13:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:13:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.949237Z", 2022-01-11T14:13:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:13:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:13:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:13:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:13:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:13:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:13:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:13:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:13:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:13:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:17.411 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:13:17.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.949237_678.1 2022-01-11T14:13:22.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:22.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:22.414 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:13:22.475 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:22.475 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:22.731 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:13:22.731 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:13:22.731 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:13:22.732 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:13:22.732 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:13:22.732 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:13:22.733 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:22.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:22.733 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:13:22.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:22.733 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:13:22.734 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:22.734 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:22.734 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.948211_677.1", 2022-01-11T14:13:22.734 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:22.735 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:13:22.735 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:22.735 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.948211Z", 2022-01-11T14:13:22.735 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:22.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:22.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:22.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:22.736 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:13:22.737 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:22.737 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:22.737 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:13:22.737 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:22.738 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:22.738 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:22.738 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:13:22.738 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:22.739 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:22.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:22.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:22.739 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:13:22.739 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:22.740 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:22.740 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:22.740 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:22.740 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:22.741 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:22.741 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:22.741 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:22.741 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:22.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:22.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:22.742 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:13:22.742 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:22.742 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:22.743 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:22.743 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:22.743 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:22.743 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:22.744 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:22.744 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:22.744 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:22.744 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:22.744 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:13:22.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:22.745 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:13:22.745 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:13:22.745 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:13:22.746 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:22.746 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:13:22.747 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.948211_677.1 2022-01-11T14:13:27.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:27.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:27.750 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:13:27.807 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:27.808 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:27.972 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:13:27.973 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:13:27.973 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:13:27.973 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:13:27.974 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:13:27.974 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:13:27.974 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:27.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:27.974 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:13:27.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:27.975 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:13:27.975 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:27.975 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:27.976 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.969581_682.1", 2022-01-11T14:13:27.976 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:27.976 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:13:27.976 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:27.977 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.969581Z", 2022-01-11T14:13:27.977 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:27.977 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:27.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:27.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:27.978 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:13:27.978 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:27.978 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:27.978 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:13:27.979 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:27.979 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:27.979 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:27.979 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:13:27.979 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:27.980 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:27.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:27.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:27.980 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:13:27.981 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:27.981 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:27.981 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:27.981 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:27.981 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:27.982 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:27.982 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:27.983 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:27.983 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:27.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:27.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:27.983 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:13:27.984 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:27.984 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:27.984 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:27.985 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:27.985 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:27.985 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:27.985 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:27.985 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:27.986 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:27.986 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:27.986 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:13:27.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:13:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:13:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:13:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:27.987 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:13:27.988 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.969581_682.1 2022-01-11T14:13:32.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:32.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:32.992 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:13:33.036 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:33.037 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:33.118 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:13:33.119 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:13:33.119 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:13:33.119 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:13:33.119 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:13:33.120 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:13:33.120 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:33.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:33.121 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:13:33.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:33.121 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:13:33.122 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:33.122 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:33.122 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:13:33.123 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:33.123 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:13:33.123 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:33.123 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:13:33.123 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:33.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:33.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:33.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:33.124 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:13:33.125 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:33.125 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:33.125 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:13:33.125 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:33.126 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:33.126 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:33.126 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:13:33.126 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:33.127 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:33.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:33.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:33.127 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:13:33.127 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:33.128 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:33.128 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:33.128 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:33.128 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:33.129 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:33.129 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:33.129 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:33.129 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:33.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:33.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:33.130 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:13:33.130 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:33.130 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:33.131 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:33.131 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:33.131 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:33.131 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:33.132 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:33.132 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:33.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:33.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:33.132 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:13:33.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:33.133 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:13:33.133 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:13:33.133 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:13:33.134 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:33.134 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:13:33.135 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.965074_680.1 2022-01-11T14:13:38.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:38.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:38.138 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:13:38.182 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:38.183 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:38.240 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:13:38.240 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:13:38.241 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:13:38.241 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:13:38.241 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:13:38.241 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:13:38.242 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:38.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:38.242 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:13:38.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:38.243 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:13:38.243 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:38.243 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:38.244 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.949237_678.1", 2022-01-11T14:13:38.244 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:38.244 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:13:38.244 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:38.244 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.949237Z", 2022-01-11T14:13:38.245 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:38.245 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:38.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:38.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:38.246 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:13:38.246 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:38.246 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:38.247 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:13:38.247 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:38.247 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:38.248 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:38.248 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:13:38.248 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:38.248 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:38.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:38.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:38.249 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:13:38.249 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:38.249 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:38.250 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:38.250 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:38.250 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:38.250 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:38.251 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:38.251 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:38.251 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:38.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:38.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:38.252 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:13:38.252 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:38.252 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:38.252 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:38.253 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:38.253 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:38.253 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:38.253 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:38.254 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:13:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:13:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:13:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:13:38.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:38.256 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:13:38.257 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.949237_678.1 2022-01-11T14:13:43.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:43.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:43.260 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:13:43.326 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:43.327 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:43.681 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:13:43.681 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:13:43.681 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:13:43.682 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:13:43.682 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:13:43.682 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:13:43.682 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:43.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:43.683 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:13:43.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:43.683 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:13:43.683 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:43.684 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:43.684 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:13:43.684 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:43.684 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:13:43.685 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:43.685 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:13:43.685 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:43.685 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:43.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:43.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:43.687 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:13:43.687 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:43.687 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:43.688 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:13:43.688 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:43.688 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:43.688 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:43.688 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:13:43.689 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:43.689 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:43.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:43.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:43.690 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:13:43.690 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:43.690 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:43.691 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:43.691 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:43.691 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:43.691 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:43.692 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:43.692 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:43.692 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:43.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:43.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:43.693 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:13:43.693 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:43.693 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:43.693 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:43.694 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:43.694 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:43.694 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:43.694 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:43.694 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:43.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:43.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:43.695 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:13:43.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:43.696 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:13:43.696 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:13:43.696 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:13:43.696 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:43.696 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:13:43.698 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.952442_679.1 2022-01-11T14:13:48.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:48.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:48.701 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:13:48.745 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:48.746 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:48.853 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:13:48.854 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:13:48.854 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:13:48.854 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:13:48.854 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:13:48.855 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:13:48.855 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:48.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:48.856 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:13:48.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:48.856 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:13:48.856 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:48.857 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:48.857 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.949237_678.1", 2022-01-11T14:13:48.857 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:48.857 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:13:48.858 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:48.858 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.949237Z", 2022-01-11T14:13:48.858 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:48.858 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:48.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:48.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:48.859 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:13:48.859 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:48.859 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:48.860 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:13:48.860 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:48.860 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:48.860 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:48.861 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:13:48.861 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:48.861 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:48.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:48.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:48.862 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:13:48.862 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:48.862 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:48.862 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:48.863 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:48.863 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:48.863 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:48.863 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:13:48.865 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:48.865 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:48.865 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:48.865 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:48.866 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:48.866 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:48.866 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:48.866 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:48.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:48.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:48.867 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:13:48.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:48.868 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:13:48.868 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:13:48.868 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:13:48.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:48.868 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:13:48.870 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.949237_678.1 2022-01-11T14:13:53.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:53.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:53.873 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:13:53.917 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:53.918 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:53.986 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:13:53.986 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:13:53.986 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:13:53.987 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:13:53.987 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:13:53.987 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:13:53.987 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:53.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:53.988 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:13:53.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:53.988 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:13:53.988 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:53.989 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:53.989 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.969581_682.1", 2022-01-11T14:13:53.989 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:53.989 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:13:53.990 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:53.990 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.969581Z", 2022-01-11T14:13:53.990 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:53.990 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:53.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:53.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:53.991 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:13:53.991 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:53.992 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:53.992 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:13:53.992 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:53.993 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:53.993 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:53.993 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:13:53.993 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:53.993 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:53.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:53.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:53.994 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:13:53.994 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:53.996 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:53.996 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:53.996 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:53.996 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:53.996 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:53.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:53.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:53.997 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:13:53.997 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:53.997 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:53.998 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:53.998 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:53.998 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:53.999 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:53.999 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:53.999 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:53.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:53.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:54.000 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:13:54.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:54.000 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:13:54.000 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:13:54.000 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:13:54.001 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:54.001 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:13:54.002 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.969581_682.1 2022-01-11T14:13:59.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:59.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:13:59.006 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:13:59.054 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:13:59.055 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:13:59.132 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:13:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:13:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:13:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:13:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:13:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:13:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:59.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:59.134 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:13:59.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:59.134 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:13:59.135 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:59.135 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:59.135 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.969581_682.1", 2022-01-11T14:13:59.135 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:59.136 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:13:59.136 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:59.136 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.969581Z", 2022-01-11T14:13:59.136 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:59.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:59.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:59.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:59.137 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:13:59.137 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:59.137 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:59.138 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:13:59.138 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:59.138 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:59.138 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:13:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:59.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:13:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:59.140 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:59.141 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:13:59.142 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:13:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:13:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:13:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:13:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:13:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:13:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:13:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:13:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:13:59.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:13:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:13:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:13:59.145 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:13:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:13:59.146 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:13:59.146 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:13:59.147 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.969581_682.1 2022-01-11T14:14:04.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:04.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:04.150 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:14:04.200 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:14:04.200 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:14:04.260 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:14:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:14:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:14:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:14:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:14:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:14:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:04.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:04.263 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:14:04.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:04.263 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:14:04.263 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:04.264 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:04.264 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910391.989445_745.1", 2022-01-11T14:14:04.264 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:04.264 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:14:04.265 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:04.265 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:11.989445Z", 2022-01-11T14:14:04.265 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:04.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:04.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:04.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:04.266 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:14:04.267 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:04.267 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:04.267 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:14:04.267 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:04.268 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:04.268 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:04.268 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:14:04.269 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:04.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:04.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:04.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:04.269 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:14:04.270 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:04.270 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:04.270 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:04.271 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:04.271 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:04.271 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:04.271 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:04.272 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:04.272 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:04.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:04.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:04.273 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:14:04.273 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:04.273 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:04.273 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:04.274 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:04.274 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:04.274 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:04.274 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:04.274 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:04.275 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:04.275 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:04.275 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:14:04.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:04.276 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:14:04.276 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:14:04.276 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:14:04.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:04.276 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:14:04.278 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910391.989445_745.1 2022-01-11T14:14:09.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:09.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:09.280 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:14:09.337 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:14:09.337 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:14:09.767 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:14:09.768 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:14:09.768 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:14:09.768 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:14:09.768 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:14:09.769 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:14:09.769 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:09.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:09.769 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:14:09.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:09.770 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:14:09.770 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:09.770 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:09.771 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.203086_695.1", 2022-01-11T14:14:09.771 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:09.771 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:14:09.771 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:09.772 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.203086Z", 2022-01-11T14:14:09.772 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:09.772 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:09.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:09.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:09.773 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:14:09.773 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:09.773 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:09.773 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:14:09.774 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:09.774 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:09.774 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:09.774 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:14:09.775 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:09.775 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:09.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:09.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:09.775 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:14:09.776 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:09.776 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:09.776 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:09.776 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:09.777 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:09.777 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:09.777 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:09.777 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:09.778 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:09.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:09.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:09.778 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:14:09.779 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:09.779 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:09.779 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:09.779 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:09.779 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:09.780 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:09.780 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:09.780 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:09.780 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:09.781 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:09.781 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:14:09.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:09.781 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:14:09.781 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:14:09.782 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:14:09.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:09.782 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:14:09.783 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.203086_695.1 2022-01-11T14:14:14.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:14.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:14.787 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:14:14.845 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:14:14.845 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:14:15.012 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:14:15.012 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:14:15.013 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:14:15.013 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:14:15.013 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:14:15.013 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:14:15.014 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:15.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:15.014 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:14:15.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:15.015 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:14:15.015 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:15.015 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:15.015 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.075268_690.1", 2022-01-11T14:14:15.015 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:15.016 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:14:15.016 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:15.016 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.075268Z", 2022-01-11T14:14:15.016 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:15.017 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:15.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:15.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:15.018 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:14:15.018 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:15.019 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:15.019 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:14:15.019 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:15.019 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:15.019 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:15.020 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:14:15.020 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:15.020 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:15.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:15.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:15.021 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:14:15.021 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:15.021 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:15.022 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:15.022 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:15.022 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:15.022 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:15.023 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:15.023 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:15.023 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:15.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:15.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:15.024 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:14:15.024 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:15.024 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:15.025 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:15.025 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:15.025 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:15.025 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:15.025 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:15.026 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:15.026 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:15.026 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:15.027 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:14:15.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:15.027 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:14:15.027 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:14:15.028 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:14:15.028 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:15.028 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:14:15.029 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.075268_690.1 2022-01-11T14:14:20.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:20.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:20.032 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:14:20.077 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:14:20.077 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:14:20.139 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:14:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:14:20.140 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:14:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:14:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:14:20.141 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:14:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:20.142 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:14:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:14:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:20.143 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:20.144 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.285437_708.1", 2022-01-11T14:14:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:14:20.145 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.285437Z", 2022-01-11T14:14:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:20.146 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:14:20.147 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:14:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:20.148 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:14:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:20.149 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:20.150 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:14:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:20.151 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:20.152 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:20.153 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:14:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:20.154 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:20.155 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:20.156 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:14:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:14:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:14:20.157 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:14:20.158 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:20.158 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:14:20.160 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.285437_708.1 2022-01-11T14:14:25.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:25.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:25.163 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:14:25.213 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:14:25.214 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:14:25.289 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:14:25.328 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:14:25.328 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:14:25.329 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:14:25.329 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:14:25.329 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:14:25.329 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:25.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:25.330 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:14:25.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:25.330 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:14:25.331 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:25.331 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:25.331 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.138030_693.1", 2022-01-11T14:14:25.331 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:25.332 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:14:25.332 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:25.332 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.138030Z", 2022-01-11T14:14:25.332 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:25.333 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:25.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:25.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:25.333 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:14:25.334 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:25.334 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:25.334 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:14:25.334 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:25.335 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:25.335 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:25.335 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:14:25.335 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:25.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:25.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:25.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:25.336 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:14:25.336 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:25.337 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:25.337 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:25.337 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:25.337 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:25.338 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:25.338 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:25.338 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:25.338 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:25.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:25.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:25.339 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:14:25.339 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:25.340 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:25.340 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:25.340 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:25.340 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:25.341 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:25.341 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:25.341 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:25.341 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:25.342 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:25.342 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:14:25.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:25.343 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:14:25.343 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:14:25.343 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:14:25.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:25.344 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:14:25.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.138030_693.1 2022-01-11T14:14:30.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:30.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:30.348 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:14:30.433 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:14:30.433 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:14:30.766 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:14:30.766 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:14:30.767 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:14:30.767 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:14:30.767 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:14:30.767 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:14:30.768 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:30.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:30.768 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:14:30.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:30.769 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:14:30.769 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:30.769 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:30.769 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:14:30.769 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:30.770 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:14:30.770 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:30.770 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:14:30.770 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:30.771 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:30.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:30.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:30.771 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:14:30.771 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:30.772 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:30.772 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:14:30.772 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:30.772 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:30.772 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:30.773 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:14:30.773 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:30.773 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:30.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:30.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:30.774 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:14:30.774 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:30.774 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:30.774 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:30.775 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:30.775 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:30.775 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:30.775 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:30.776 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:30.776 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:30.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:30.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:30.776 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:14:30.777 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:30.777 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:30.777 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:30.777 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:30.778 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:30.778 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:30.778 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:30.778 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:30.779 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:30.779 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:30.779 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:14:30.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:30.779 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:14:30.780 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:14:30.780 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:14:30.780 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:30.780 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:14:30.782 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.965074_680.1 2022-01-11T14:14:35.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:35.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:35.784 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:14:35.847 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:14:35.847 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:14:35.958 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:14:35.959 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:14:35.959 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:14:35.960 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:14:35.960 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:14:35.961 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:14:35.961 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:35.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:35.962 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:14:35.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:35.963 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:14:35.964 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:35.964 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:35.964 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.969581_682.1", 2022-01-11T14:14:35.965 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:35.965 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:14:35.966 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:35.966 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.969581Z", 2022-01-11T14:14:35.967 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:35.967 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:35.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:35.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:35.969 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:14:35.969 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:35.970 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:35.970 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:14:35.970 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:35.971 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:35.971 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:35.972 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:14:35.972 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:35.973 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:35.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:35.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:35.974 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:14:35.974 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:35.975 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:35.975 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:35.976 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:35.976 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:35.977 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:35.977 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:35.978 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:35.978 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:35.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:35.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:35.980 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:14:35.980 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:35.981 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:35.981 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:35.982 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:35.982 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:35.983 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:35.983 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:35.983 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:35.984 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:35.985 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:35.985 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:14:35.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:35.986 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:14:35.987 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:14:35.987 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:14:35.987 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:35.988 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:14:35.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.969581_682.1 2022-01-11T14:14:40.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:40.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:40.993 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:14:41.042 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:14:41.042 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:14:41.118 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:14:41.119 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:14:41.119 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:14:41.119 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:14:41.120 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:14:41.120 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:14:41.120 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:41.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:41.120 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:14:41.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:41.121 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:14:41.121 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:41.121 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:41.122 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.002077_686.1", 2022-01-11T14:14:41.122 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:41.122 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:14:41.123 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:41.123 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.002077Z", 2022-01-11T14:14:41.123 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:41.123 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:41.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:41.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:41.124 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:14:41.124 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:41.125 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:41.125 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:14:41.125 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:41.125 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:41.125 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:41.126 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:14:41.126 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:41.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:41.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:41.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:41.127 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:14:41.127 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:41.127 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:41.127 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:41.128 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:41.128 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:41.128 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:41.128 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:41.129 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:41.131 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:41.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:41.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:41.131 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:14:41.132 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:41.132 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:41.132 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:41.132 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:41.133 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:41.133 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:41.133 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:41.133 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:41.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:41.134 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:41.134 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:14:41.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:41.134 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:14:41.135 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:14:41.135 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:14:41.135 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:41.135 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:14:41.137 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.002077_686.1 2022-01-11T14:14:46.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:46.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:46.139 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:14:46.189 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:14:46.189 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:14:46.246 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:14:46.247 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:14:46.247 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:14:46.247 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:14:46.248 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:14:46.248 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:14:46.248 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:46.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:46.249 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:14:46.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:46.249 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:14:46.249 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:46.249 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:46.250 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.230925_700.1", 2022-01-11T14:14:46.250 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:46.250 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:14:46.251 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:46.251 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.230925Z", 2022-01-11T14:14:46.251 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:46.251 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:46.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:46.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:46.252 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:14:46.252 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:46.252 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:46.253 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:14:46.253 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:46.253 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:46.253 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:46.254 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:14:46.254 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:46.254 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:46.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:46.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:46.255 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:14:46.255 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:46.255 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:46.255 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:46.256 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:46.256 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:46.256 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:46.256 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:46.257 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:46.257 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:46.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:46.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:46.258 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:14:46.258 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:46.258 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:46.258 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:46.259 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:46.259 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:46.259 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:46.259 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:46.260 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:46.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:46.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:46.260 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:14:46.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:46.261 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:14:46.261 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:14:46.261 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:14:46.261 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:46.262 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:14:46.263 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.230925_700.1 2022-01-11T14:14:51.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:51.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:51.267 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:14:51.317 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:14:51.317 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:14:51.575 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:14:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:14:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:14:51.576 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:14:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:14:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:14:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:51.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:14:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:14:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:51.578 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:14:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:14:51.579 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:14:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:51.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:14:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:51.581 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:14:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:51.582 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:14:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:51.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:14:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:51.584 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:51.585 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:51.586 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:51.587 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:14:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:51.588 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:51.589 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:51.590 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:14:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:14:51.591 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:14:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:14:51.592 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:51.592 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:14:51.593 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.952442_679.1 2022-01-11T14:14:56.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:56.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:14:56.596 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:14:56.656 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:14:56.656 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:14:56.838 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:14:56.839 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:14:56.839 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:14:56.839 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:14:56.839 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:14:56.840 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:14:56.840 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:56.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:56.840 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:14:56.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:56.841 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:14:56.841 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:56.842 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:56.842 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.004662_687.1", 2022-01-11T14:14:56.842 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:56.843 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:14:56.843 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:56.843 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.004662Z", 2022-01-11T14:14:56.843 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:56.843 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:56.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:56.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:56.844 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:14:56.844 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:56.845 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:56.845 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:14:56.845 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:56.846 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:56.846 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:56.847 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:14:56.847 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:56.848 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:56.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:56.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:56.849 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:14:56.849 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:56.850 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:56.851 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:56.851 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:56.851 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:56.851 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:56.852 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:56.852 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:56.852 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:56.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:56.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:14:56.853 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:14:56.853 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:14:56.853 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:14:56.854 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:14:56.854 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:14:56.854 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:14:56.854 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:14:56.854 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:14:56.855 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:14:56.855 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:56.855 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:56.855 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:14:56.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:14:56.856 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:14:56.856 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:14:56.856 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:14:56.856 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:14:56.857 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:14:56.857 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.004662_687.1 2022-01-11T14:15:01.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:01.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:01.860 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:15:01.903 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:15:01.904 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:15:01.978 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:15:01.978 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:15:01.978 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:15:01.979 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:15:01.979 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:15:01.979 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:15:01.979 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:01.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:01.980 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:15:01.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:01.981 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:15:01.981 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:01.981 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:01.981 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:15:01.981 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:01.982 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:15:01.982 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:01.982 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:15:01.982 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:01.983 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:01.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:01.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:01.983 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:15:01.984 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:01.984 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:01.984 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:15:01.984 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:01.985 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:01.985 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:01.985 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:15:01.985 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:01.986 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:01.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:01.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:01.986 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:15:01.987 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:01.987 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:01.987 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:01.987 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:01.988 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:01.988 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:01.988 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:01.988 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:01.988 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:01.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:01.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:01.989 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:15:01.989 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:01.990 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:01.990 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:01.990 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:01.990 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:01.991 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:01.991 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:01.991 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:01.991 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:01.992 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:01.992 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:15:01.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:01.992 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:15:01.993 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:15:01.993 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:15:01.993 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:01.993 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:15:01.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.965074_680.1 2022-01-11T14:15:06.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:06.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:06.998 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:15:07.047 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:15:07.047 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:15:07.130 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:15:07.130 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:15:07.130 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:15:07.131 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:15:07.131 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:15:07.131 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:15:07.131 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:07.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:07.132 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:15:07.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:07.132 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:15:07.132 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:07.133 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:07.133 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.138030_693.1", 2022-01-11T14:15:07.133 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:07.133 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:15:07.134 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:07.134 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.138030Z", 2022-01-11T14:15:07.134 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:07.135 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:07.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:07.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:07.135 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:15:07.136 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:07.136 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:07.136 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:15:07.136 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:07.136 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:07.137 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:07.137 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:15:07.137 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:07.137 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:07.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:07.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:07.138 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:15:07.138 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:07.138 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:07.139 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:07.139 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:07.139 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:07.139 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:07.139 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:07.140 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:07.140 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:07.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:07.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:07.141 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:15:07.141 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:07.141 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:07.141 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:07.141 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:07.142 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:07.142 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:07.142 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:07.142 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:07.143 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:07.143 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:07.143 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:15:07.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:07.144 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:15:07.144 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:15:07.144 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:15:07.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:07.144 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:15:07.146 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.138030_693.1 2022-01-11T14:15:12.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:12.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:12.149 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:15:12.228 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:15:12.229 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:15:12.459 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:15:12.459 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:15:12.459 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:15:12.460 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:15:12.460 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:15:12.460 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:15:12.460 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:12.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:12.461 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:15:12.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:12.461 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:15:12.461 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:12.462 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:12.462 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.002077_686.1", 2022-01-11T14:15:12.462 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:12.462 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:15:12.463 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:12.463 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.002077Z", 2022-01-11T14:15:12.463 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:12.463 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:12.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:12.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:12.464 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:15:12.464 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:12.464 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:12.465 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:15:12.465 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:12.465 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:12.465 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:12.466 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:15:12.466 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:12.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:12.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:12.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:12.467 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:15:12.467 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:12.467 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:12.467 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:12.468 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:12.468 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:12.468 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:12.468 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:12.469 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:12.469 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:12.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:12.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:12.470 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:15:12.470 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:12.470 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:12.470 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:12.470 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:12.471 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:12.471 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:12.471 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:12.471 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:12.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:12.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:12.472 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:15:12.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:12.473 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:15:12.473 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:15:12.473 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:15:12.473 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:12.473 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:15:12.475 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.002077_686.1 2022-01-11T14:15:17.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:17.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:17.477 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:15:17.549 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:15:17.550 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:15:17.719 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:15:17.719 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:15:17.720 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:15:17.720 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:15:17.720 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:15:17.720 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:15:17.721 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:17.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:17.721 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:15:17.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:17.721 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:15:17.722 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:17.722 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:17.722 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.230925_700.1", 2022-01-11T14:15:17.722 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:17.723 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:15:17.723 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:17.723 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.230925Z", 2022-01-11T14:15:17.723 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:17.723 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:17.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:17.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:17.724 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:15:17.724 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:17.725 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:17.725 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:15:17.725 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:17.725 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:17.725 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:17.726 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:15:17.726 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:17.726 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:17.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:17.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:17.727 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:15:17.727 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:17.727 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:17.727 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:17.728 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:17.728 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:17.728 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:17.728 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:17.728 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:17.729 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:17.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:17.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:17.729 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:15:17.730 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:17.730 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:17.730 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:17.730 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:17.730 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:17.731 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:17.731 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:17.731 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:17.731 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:17.731 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:17.732 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:15:17.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:17.732 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:15:17.732 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:15:17.733 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:15:17.733 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:17.733 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:15:17.734 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.230925_700.1 2022-01-11T14:15:22.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:22.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:22.737 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:15:22.786 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:15:22.787 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:15:22.875 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:15:22.875 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:15:22.876 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:15:22.876 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:15:22.876 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:15:22.876 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:15:22.876 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:22.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:22.877 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:15:22.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:22.878 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:15:22.878 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:22.878 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:22.879 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.075268_690.1", 2022-01-11T14:15:22.879 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:22.879 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:15:22.880 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:22.880 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.075268Z", 2022-01-11T14:15:22.880 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:22.880 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:22.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:22.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:22.881 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:15:22.881 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:22.882 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:22.882 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:15:22.882 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:22.882 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:22.882 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:22.883 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:15:22.883 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:22.883 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:22.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:22.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:22.884 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:15:22.884 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:22.884 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:22.885 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:22.885 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:22.885 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:22.885 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:22.885 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:22.886 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:22.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:22.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:22.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:22.887 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:15:22.887 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:22.887 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:22.887 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:22.888 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:22.888 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:22.888 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:22.888 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:22.889 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:22.889 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:22.890 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:22.890 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:15:22.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:22.890 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:15:22.891 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:15:22.891 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:15:22.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:22.891 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:15:22.893 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.075268_690.1 2022-01-11T14:15:27.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:27.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:27.895 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:15:27.945 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:15:27.945 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:15:28.009 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:15:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:15:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:15:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:15:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:15:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:15:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:15:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:15:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.969581_682.1", 2022-01-11T14:15:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:15:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.969581Z", 2022-01-11T14:15:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:15:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:15:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:15:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:15:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:15:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:15:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:15:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:15:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:15:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:28.024 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:15:28.026 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.969581_682.1 2022-01-11T14:15:33.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:33.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:33.029 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:15:33.119 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:15:33.120 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:15:33.456 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:15:33.457 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:15:33.457 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:15:33.458 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:15:33.458 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:15:33.458 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:15:33.458 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:33.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:33.459 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:15:33.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:33.459 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:15:33.460 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:33.460 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:33.460 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.075268_690.1", 2022-01-11T14:15:33.460 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:33.461 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:15:33.461 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:33.461 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.075268Z", 2022-01-11T14:15:33.461 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:33.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:33.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:33.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:33.462 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:15:33.462 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:33.463 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:33.463 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:15:33.463 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:33.463 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:33.464 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:33.464 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:15:33.464 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:33.464 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:33.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:33.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:33.465 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:15:33.465 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:33.465 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:33.466 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:33.466 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:33.466 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:33.466 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:33.467 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:33.467 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:33.467 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:33.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:33.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:33.468 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:15:33.468 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:33.468 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:33.469 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:33.469 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:33.469 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:33.469 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:33.469 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:33.470 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:33.470 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:33.470 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:33.470 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:15:33.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:33.471 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:15:33.471 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:15:33.471 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:15:33.471 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:33.472 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:15:33.473 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.075268_690.1 2022-01-11T14:15:38.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:38.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:38.476 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:15:38.541 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:15:38.542 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:15:38.697 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:15:38.698 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:15:38.698 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:15:38.698 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:15:38.698 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:15:38.699 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:15:38.699 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:38.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:38.700 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:15:38.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:38.700 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:15:38.700 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:38.701 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:38.701 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.198209_694.1", 2022-01-11T14:15:38.701 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:38.702 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:15:38.702 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:38.702 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.198209Z", 2022-01-11T14:15:38.702 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:38.703 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:38.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:38.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:38.703 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:15:38.703 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:38.704 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:38.704 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:15:38.704 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:38.704 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:38.705 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:38.705 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:15:38.705 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:38.705 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:38.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:38.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:38.706 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:15:38.707 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:38.707 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:38.707 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:38.707 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:38.708 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:38.708 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:38.708 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:38.708 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:38.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:38.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:38.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:38.709 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:15:38.710 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:38.710 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:38.710 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:38.710 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:38.711 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:38.711 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:38.711 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:38.711 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:38.712 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:38.712 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:38.712 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:15:38.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:38.713 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:15:38.713 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:15:38.713 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:15:38.714 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:38.714 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:15:38.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.198209_694.1 2022-01-11T14:15:43.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:43.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:43.718 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:15:43.762 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:15:43.762 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:15:43.837 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:15:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:15:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:15:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:15:43.838 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:15:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:15:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:15:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:15:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:43.840 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.020042_688.1", 2022-01-11T14:15:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:15:43.841 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.020042Z", 2022-01-11T14:15:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:43.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:15:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:43.843 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:15:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:43.844 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:15:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:43.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:15:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:43.846 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:43.847 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:43.848 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:15:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:43.849 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:43.850 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:15:43.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:15:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:15:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:15:43.852 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:43.852 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:15:43.854 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.020042_688.1 2022-01-11T14:15:48.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:48.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:48.857 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:15:48.906 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:15:48.907 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:15:48.974 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:15:48.974 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:15:48.974 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:15:48.975 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:15:48.975 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:15:48.975 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:15:48.976 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:48.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:48.976 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:15:48.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:48.977 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:15:48.977 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:48.977 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:48.977 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.230925_700.1", 2022-01-11T14:15:48.978 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:48.978 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:15:48.978 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:48.978 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.230925Z", 2022-01-11T14:15:48.979 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:48.979 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:48.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:48.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:48.979 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:15:48.980 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:48.980 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:48.980 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:15:48.980 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:48.981 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:48.981 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:48.981 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:15:48.981 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:48.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:48.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:48.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:48.982 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:15:48.983 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:48.983 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:48.983 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:48.983 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:48.984 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:48.984 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:48.984 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:48.984 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:48.984 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:48.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:48.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:48.985 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:15:48.985 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:48.986 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:48.988 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:48.989 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:48.989 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:48.989 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:48.990 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:48.990 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:48.990 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:48.991 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:48.991 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:15:48.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:48.991 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:15:48.992 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:15:48.992 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:15:48.992 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:48.992 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:15:48.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.230925_700.1 2022-01-11T14:15:53.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:53.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:53.996 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:15:54.045 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:15:54.045 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:15:54.121 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:15:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:15:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:15:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:15:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:15:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:15:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:54.123 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:15:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:15:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:54.124 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.134640_692.1", 2022-01-11T14:15:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:54.125 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:15:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.134640Z", 2022-01-11T14:15:54.126 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:54.127 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:15:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:15:54.128 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:15:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:54.130 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:15:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:54.131 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:54.132 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:54.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:15:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:54.134 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:54.135 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:15:54.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:15:54.137 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:15:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:15:54.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:54.138 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:15:54.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.134640_692.1 2022-01-11T14:15:59.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:59.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:15:59.143 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:15:59.229 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:15:59.229 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:15:59.488 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:15:59.489 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:15:59.489 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:15:59.489 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:15:59.490 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:15:59.490 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:15:59.490 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:59.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:59.491 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:15:59.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:59.491 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:15:59.491 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:59.492 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:59.492 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.000883_685.1", 2022-01-11T14:15:59.492 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:59.492 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:15:59.493 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:59.493 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.000883Z", 2022-01-11T14:15:59.493 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:59.493 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:59.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:59.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:59.494 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:15:59.494 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:59.495 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:59.495 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:15:59.495 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:59.495 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:59.496 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:59.496 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:15:59.496 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:59.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:59.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:59.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:59.497 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:15:59.497 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:59.497 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:59.498 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:59.498 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:59.498 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:59.498 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:59.498 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:59.499 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:59.499 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:59.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:59.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:15:59.500 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:15:59.500 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:15:59.500 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:15:59.500 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:15:59.500 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:15:59.501 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:15:59.501 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:15:59.501 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:15:59.501 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:15:59.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:59.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:59.502 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:15:59.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:15:59.502 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:15:59.503 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:15:59.503 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:15:59.503 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:15:59.503 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:15:59.505 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.000883_685.1 2022-01-11T14:16:04.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:04.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:04.507 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:16:04.563 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:16:04.564 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:16:04.674 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:16:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:16:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:16:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:16:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:16:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:16:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:16:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:16:04.677 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:04.677 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:04.677 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:16:04.677 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:04.678 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:16:04.678 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:04.678 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:16:04.678 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:16:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:16:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:16:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:16:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:16:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:16:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:16:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:16:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:16:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:04.690 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:16:04.691 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.952442_679.1 2022-01-11T14:16:09.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:09.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:09.693 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:16:09.738 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:16:09.738 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:16:09.797 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:16:09.797 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:16:09.798 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:16:09.798 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:16:09.798 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:16:09.799 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:16:09.799 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:09.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:09.799 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:16:09.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:09.800 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:16:09.800 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:09.800 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:09.800 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.973179_683.1", 2022-01-11T14:16:09.801 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:09.801 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:16:09.801 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:09.801 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.973179Z", 2022-01-11T14:16:09.801 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:09.802 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:09.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:09.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:09.802 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:16:09.803 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:09.803 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:09.803 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:16:09.803 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:09.804 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:09.804 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:09.804 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:16:09.804 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:09.804 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:09.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:09.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:09.805 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:16:09.805 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:09.806 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:09.806 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:09.806 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:09.806 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:09.807 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:09.807 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:09.807 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:09.807 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:09.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:09.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:09.808 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:16:09.808 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:09.808 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:09.809 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:09.809 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:09.809 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:09.809 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:09.809 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:09.810 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:09.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:09.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:09.810 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:16:09.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:09.811 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:16:09.811 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:16:09.811 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:16:09.811 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:09.812 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:16:09.813 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.973179_683.1 2022-01-11T14:16:14.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:14.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:14.815 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:16:14.863 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:16:14.863 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:16:14.932 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:16:14.932 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:16:14.933 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:16:14.933 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:16:14.933 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:16:14.933 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:16:14.933 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:14.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:14.934 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:16:14.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:14.935 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:16:14.935 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:14.935 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:14.935 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.968115_681.1", 2022-01-11T14:16:14.935 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:14.936 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:16:14.936 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:14.936 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.968115Z", 2022-01-11T14:16:14.936 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:14.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:14.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:14.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:14.937 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:16:14.937 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:14.938 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:14.938 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:16:14.938 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:14.939 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:14.940 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:14.940 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:16:14.940 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:14.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:14.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:14.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:14.941 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:16:14.941 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:14.941 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:14.942 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:14.942 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:14.942 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:14.942 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:14.943 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:14.943 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:14.943 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:14.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:14.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:14.944 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:16:14.944 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:14.944 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:14.944 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:14.944 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:14.945 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:14.945 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:14.945 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:14.945 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:14.946 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:14.946 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:14.946 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:16:14.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:14.947 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:16:14.947 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:16:14.947 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:16:14.947 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:14.947 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:16:14.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.968115_681.1 2022-01-11T14:16:19.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:19.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:19.952 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:16:20.058 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:16:20.058 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:16:20.358 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:16:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:16:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:16:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:16:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:16:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:16:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:16:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:16:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.020042_688.1", 2022-01-11T14:16:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:16:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.020042Z", 2022-01-11T14:16:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:16:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:16:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:16:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:16:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:16:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:16:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:20.374 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:16:20.374 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:16:20.374 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:16:20.375 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:20.375 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:16:20.377 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.020042_688.1 2022-01-11T14:16:25.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:25.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:25.380 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:16:25.457 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:16:25.458 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:16:25.579 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:16:25.579 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:16:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:16:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:16:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:16:25.580 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:16:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:16:25.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:16:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.998061_684.1", 2022-01-11T14:16:25.582 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:16:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.998061Z", 2022-01-11T14:16:25.583 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:25.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:25.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:25.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:25.584 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:16:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:16:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:25.585 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:16:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:25.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:16:25.587 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:25.588 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:25.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:16:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:25.590 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:25.591 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:25.592 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:16:25.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:25.593 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:16:25.593 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:16:25.593 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:16:25.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:25.594 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:16:25.595 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.998061_684.1 2022-01-11T14:16:30.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:30.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:30.599 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:16:30.644 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:16:30.644 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:16:30.725 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:16:30.725 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:16:30.725 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:16:30.726 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:16:30.726 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:16:30.726 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:16:30.727 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:30.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:30.728 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:16:30.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:30.729 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:16:30.729 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:30.729 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:30.730 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.020042_688.1", 2022-01-11T14:16:30.730 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:30.731 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:16:30.731 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:30.732 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.020042Z", 2022-01-11T14:16:30.732 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:30.733 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:30.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:30.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:30.734 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:16:30.734 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:30.735 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:30.735 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:16:30.736 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:30.736 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:30.736 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:30.737 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:16:30.737 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:30.738 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:30.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:30.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:30.739 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:16:30.739 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:30.740 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:30.740 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:30.741 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:30.741 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:30.742 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:30.742 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:30.743 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:30.743 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:30.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:30.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:30.744 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:16:30.745 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:30.745 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:30.745 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:30.746 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:30.746 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:30.747 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:30.747 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:30.748 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:30.748 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:30.749 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:30.749 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:16:30.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:30.750 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:16:30.750 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:16:30.751 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:16:30.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:30.752 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:16:30.754 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.020042_688.1 2022-01-11T14:16:35.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:35.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:35.758 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:16:35.803 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:16:35.804 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:16:35.862 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:16:35.862 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:16:35.863 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:16:35.863 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:16:35.863 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:16:35.863 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:16:35.863 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:35.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:35.864 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:16:35.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:35.864 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:16:35.865 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:35.865 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:35.865 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.138030_693.1", 2022-01-11T14:16:35.865 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:35.866 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:16:35.866 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:35.866 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.138030Z", 2022-01-11T14:16:35.866 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:35.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:35.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:35.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:35.868 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:16:35.868 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:35.868 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:35.868 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:16:35.868 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:35.869 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:35.869 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:35.870 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:16:35.870 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:35.870 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:35.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:35.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:35.871 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:16:35.871 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:35.871 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:35.872 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:35.872 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:35.872 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:35.872 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:35.873 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:35.873 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:35.873 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:35.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:35.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:35.874 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:16:35.874 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:35.874 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:35.874 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:35.875 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:35.875 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:35.875 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:35.875 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:35.876 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:35.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:35.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:35.876 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:16:35.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:35.877 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:16:35.877 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:16:35.877 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:16:35.877 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:35.878 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:16:35.879 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.138030_693.1 2022-01-11T14:16:40.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:40.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:40.882 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:16:40.971 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:16:40.971 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:16:41.216 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:16:41.216 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:16:41.216 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:16:41.217 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:16:41.217 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:16:41.217 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:16:41.217 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:41.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:41.218 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:16:41.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:41.218 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:16:41.219 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:41.219 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:41.219 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:16:41.220 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:41.220 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:16:41.220 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:41.220 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:16:41.221 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:41.221 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:41.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:41.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:41.222 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:16:41.222 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:41.222 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:41.223 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:16:41.223 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:41.223 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:41.223 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:41.224 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:16:41.224 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:41.224 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:41.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:41.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:41.225 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:16:41.225 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:41.225 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:41.226 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:41.226 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:41.226 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:41.226 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:41.227 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:41.227 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:41.227 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:41.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:41.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:41.228 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:16:41.228 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:41.228 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:41.228 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:41.229 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:41.229 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:41.229 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:41.229 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:41.230 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:41.230 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:41.230 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:41.230 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:16:41.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:41.231 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:16:41.231 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:16:41.231 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:16:41.231 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:41.232 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:16:41.233 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.952442_679.1 2022-01-11T14:16:46.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:46.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:46.237 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:16:46.302 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:16:46.302 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:16:46.381 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:16:46.381 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:16:46.381 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:16:46.381 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:16:46.382 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:16:46.382 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:16:46.382 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:46.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:46.383 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:16:46.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:46.383 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:16:46.383 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:46.384 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:46.384 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.134640_692.1", 2022-01-11T14:16:46.384 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:46.384 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:16:46.385 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:46.385 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.134640Z", 2022-01-11T14:16:46.385 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:46.385 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:46.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:46.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:46.386 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:16:46.386 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:46.386 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:46.387 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:16:46.387 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:46.387 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:46.387 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:46.388 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:16:46.388 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:46.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:46.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:46.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:46.389 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:16:46.389 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:46.389 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:46.389 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:46.390 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:46.390 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:46.390 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:46.390 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:46.391 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:46.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:46.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:46.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:46.391 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:16:46.392 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:46.392 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:46.392 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:46.392 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:46.392 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:46.393 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:46.393 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:46.393 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:46.393 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:46.394 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:46.394 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:16:46.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:46.394 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:16:46.395 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:16:46.395 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:16:46.395 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:46.395 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:16:46.396 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.134640_692.1 2022-01-11T14:16:51.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:51.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:51.399 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:16:51.443 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:16:51.444 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:16:51.518 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:16:51.518 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:16:51.518 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:16:51.519 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:16:51.519 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:16:51.520 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:16:51.520 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:51.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:51.521 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:16:51.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:51.521 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:16:51.521 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:51.522 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:51.522 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.204054_696.1", 2022-01-11T14:16:51.522 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:51.522 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:16:51.523 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:51.523 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.204054Z", 2022-01-11T14:16:51.523 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:51.523 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:51.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:51.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:51.524 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:16:51.524 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:51.525 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:51.525 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:16:51.525 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:51.525 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:51.526 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:51.526 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:16:51.526 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:51.526 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:51.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:51.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:51.527 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:16:51.527 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:51.528 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:51.528 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:51.528 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:51.528 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:51.529 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:51.529 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:51.529 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:51.529 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:51.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:51.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:51.530 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:16:51.530 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:51.530 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:51.531 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:51.531 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:51.531 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:51.531 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:51.532 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:51.532 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:51.532 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:51.532 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:51.533 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:16:51.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:51.533 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:16:51.533 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:16:51.533 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:16:51.534 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:51.534 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:16:51.535 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.204054_696.1 2022-01-11T14:16:56.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:56.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:16:56.539 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:16:56.583 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:16:56.583 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:16:56.637 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:16:56.638 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:16:56.638 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:16:56.638 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:16:56.639 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:16:56.639 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:16:56.639 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:56.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:56.640 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:16:56.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:56.640 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:16:56.640 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:56.641 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:56.641 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.969581_682.1", 2022-01-11T14:16:56.641 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:56.641 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:16:56.641 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:56.642 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.969581Z", 2022-01-11T14:16:56.642 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:56.642 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:56.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:56.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:56.643 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:16:56.643 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:56.644 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:56.644 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:16:56.644 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:56.644 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:56.645 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:56.645 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:16:56.645 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:56.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:56.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:56.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:56.646 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:16:56.646 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:56.646 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:56.647 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:56.647 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:56.647 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:56.647 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:56.648 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:56.648 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:56.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:56.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:56.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:16:56.649 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:16:56.649 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:16:56.649 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:16:56.649 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:16:56.650 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:16:56.650 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:16:56.650 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:16:56.650 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:16:56.650 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:16:56.651 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:56.651 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:56.651 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:16:56.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:16:56.652 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:16:56.652 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:16:56.652 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:16:56.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:16:56.652 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:16:56.654 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.969581_682.1 2022-01-11T14:17:01.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:01.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:01.657 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:17:01.702 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:17:01.703 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:17:01.907 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:17:01.908 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:17:01.908 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:17:01.908 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:17:01.908 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:17:01.909 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:17:01.909 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:01.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:01.909 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:17:01.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:01.910 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:17:01.910 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:01.911 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:01.911 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.002077_686.1", 2022-01-11T14:17:01.911 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:01.911 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:17:01.912 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:01.912 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.002077Z", 2022-01-11T14:17:01.912 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:01.912 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:01.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:01.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:01.913 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:17:01.914 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:01.914 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:01.914 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:17:01.914 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:01.915 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:01.915 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:01.915 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:17:01.915 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:01.916 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:01.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:01.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:01.916 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:17:01.916 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:01.917 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:01.917 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:01.917 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:01.917 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:01.918 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:01.918 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:01.918 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:01.918 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:01.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:01.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:01.919 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:17:01.919 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:01.919 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:01.920 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:01.920 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:01.920 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:01.920 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:01.921 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:01.921 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:01.921 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:01.921 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:01.921 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:17:01.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:01.922 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:17:01.922 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:17:01.922 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:17:01.923 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:01.923 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:17:01.925 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.002077_686.1 2022-01-11T14:17:06.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:06.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:06.927 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:17:07.004 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:17:07.004 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:17:07.117 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:17:07.117 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:17:07.117 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:17:07.117 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:17:07.118 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:17:07.118 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:17:07.118 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:07.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:07.119 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:17:07.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:07.119 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:17:07.119 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:07.120 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:07.120 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.000883_685.1", 2022-01-11T14:17:07.120 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:07.120 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:17:07.121 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:07.121 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.000883Z", 2022-01-11T14:17:07.121 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:07.121 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:07.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:07.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:07.122 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:17:07.122 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:07.122 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:07.123 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:17:07.123 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:07.123 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:07.123 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:07.124 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:17:07.124 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:07.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:07.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:07.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:07.125 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:17:07.125 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:07.125 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:07.126 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:07.126 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:07.126 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:07.126 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:07.126 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:07.127 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:07.127 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:07.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:07.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:07.128 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:17:07.128 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:07.128 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:07.128 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:07.129 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:07.129 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:07.129 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:07.129 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:07.129 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:07.130 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:07.130 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:07.130 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:17:07.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:07.131 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:17:07.131 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:17:07.131 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:17:07.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:07.132 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:17:07.132 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.000883_685.1 2022-01-11T14:17:12.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:12.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:12.135 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:17:12.178 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:17:12.179 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:17:12.251 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:17:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:17:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:17:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:17:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:17:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:17:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:17:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:17:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.004662_687.1", 2022-01-11T14:17:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:17:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.004662Z", 2022-01-11T14:17:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:17:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:17:12.258 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:12.258 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:12.258 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:12.258 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:17:12.258 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:12.259 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:12.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:12.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:12.259 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:17:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:17:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:12.263 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:12.263 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:12.263 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:12.263 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:12.263 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:12.265 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:17:12.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:12.265 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:17:12.265 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:17:12.265 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:17:12.266 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:12.266 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:17:12.267 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.004662_687.1 2022-01-11T14:17:17.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:17.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:17.270 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:17:17.321 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:17:17.321 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:17:17.397 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:17:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:17:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:17:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:17:17.398 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:17:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:17:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:17.399 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:17:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:17:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:17.400 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:17:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:17:17.401 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:17:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:17.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:17:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:17.403 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:17:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:17.404 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:17:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:17.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:17:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:17.406 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:17.407 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:17.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:17:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:17.409 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:17.410 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:17:17.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:17.412 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:17:17.412 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:17:17.412 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:17:17.412 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:17.412 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:17:17.414 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.965074_680.1 2022-01-11T14:17:22.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:22.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:22.416 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:17:22.461 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:17:22.461 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:17:22.516 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:17:22.517 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:17:22.517 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:17:22.517 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:17:22.518 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:17:22.518 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:17:22.518 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:22.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:22.519 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:17:22.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:22.519 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:17:22.519 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:22.519 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:22.520 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.973179_683.1", 2022-01-11T14:17:22.520 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:22.520 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:17:22.520 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:22.521 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.973179Z", 2022-01-11T14:17:22.521 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:22.521 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:22.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:22.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:22.522 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:17:22.522 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:22.522 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:22.522 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:17:22.523 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:22.523 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:22.523 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:22.523 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:17:22.524 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:22.524 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:22.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:22.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:22.525 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:17:22.525 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:22.525 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:22.525 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:22.525 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:22.526 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:22.526 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:22.526 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:22.526 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:22.527 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:22.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:22.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:22.527 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:17:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:22.529 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:22.529 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:22.529 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:22.529 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:22.530 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:22.530 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:17:22.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:22.530 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:17:22.530 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:17:22.531 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:17:22.531 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:22.531 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:17:22.532 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.973179_683.1 2022-01-11T14:17:27.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:27.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:27.536 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:17:27.629 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:17:27.630 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:17:27.915 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:17:27.915 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:17:27.916 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:17:27.916 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:17:27.916 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:17:27.916 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:17:27.916 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:27.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:27.917 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:17:27.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:27.917 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:17:27.918 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:27.918 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:27.918 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:17:27.918 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:27.918 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:17:27.919 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:27.919 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:17:27.919 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:27.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:27.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:27.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:27.920 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:17:27.920 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:27.920 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:27.921 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:17:27.921 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:27.921 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:27.921 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:27.922 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:17:27.922 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:27.922 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:27.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:27.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:27.923 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:17:27.923 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:27.923 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:27.923 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:27.924 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:27.924 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:27.924 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:27.924 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:27.924 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:27.925 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:27.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:27.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:27.925 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:17:27.925 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:27.926 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:27.926 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:27.926 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:27.926 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:27.927 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:27.927 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:27.927 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:27.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:27.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:27.928 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:17:27.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:27.928 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:17:27.928 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:17:27.929 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:17:27.929 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:27.929 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:17:27.929 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.952442_679.1 2022-01-11T14:17:32.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:32.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:32.932 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:17:32.995 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:17:32.995 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:17:33.074 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:17:33.074 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:17:33.075 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:17:33.075 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:17:33.075 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:17:33.076 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:17:33.076 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:33.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:33.076 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:17:33.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:33.077 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:17:33.077 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:33.077 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:33.077 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:17:33.078 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:33.078 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:17:33.078 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:33.078 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:17:33.079 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:33.079 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:33.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:33.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:33.080 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:17:33.080 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:33.080 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:33.080 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:17:33.080 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:33.081 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:33.081 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:33.081 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:17:33.081 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:33.082 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:33.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:33.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:33.082 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:17:33.083 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:33.083 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:33.083 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:33.083 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:33.084 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:33.084 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:33.084 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:33.084 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:33.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:33.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:33.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:33.085 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:17:33.085 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:33.086 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:33.086 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:33.086 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:33.086 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:33.087 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:33.087 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:33.087 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:33.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:33.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:33.088 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:17:33.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:33.088 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:17:33.089 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:17:33.089 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:17:33.089 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:33.089 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:17:33.091 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.952442_679.1 2022-01-11T14:17:38.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:38.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:38.094 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:17:38.136 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:17:38.137 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:17:38.212 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:17:38.213 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:17:38.213 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:17:38.213 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:17:38.214 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:17:38.214 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:17:38.214 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:38.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:38.215 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:17:38.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:38.215 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:17:38.215 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:38.215 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:38.216 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.204054_696.1", 2022-01-11T14:17:38.216 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:38.216 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:17:38.216 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:38.217 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.204054Z", 2022-01-11T14:17:38.217 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:38.217 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:38.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:38.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:38.218 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:17:38.218 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:38.218 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:38.218 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:17:38.219 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:38.219 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:38.219 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:38.220 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:17:38.220 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:38.220 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:38.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:38.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:38.221 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:17:38.221 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:38.221 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:38.221 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:38.222 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:38.222 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:38.222 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:38.222 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:38.222 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:38.223 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:38.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:38.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:38.223 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:17:38.224 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:38.224 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:38.224 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:38.224 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:38.224 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:38.225 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:38.225 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:38.225 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:38.225 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:38.226 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:38.226 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:17:38.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:38.226 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:17:38.227 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:17:38.227 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:17:38.227 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:38.227 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:17:38.228 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.204054_696.1 2022-01-11T14:17:43.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:43.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:43.231 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:17:43.276 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:17:43.276 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:17:43.351 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:17:43.352 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:17:43.352 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:17:43.352 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:17:43.352 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:17:43.353 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:17:43.353 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:43.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:43.353 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:17:43.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:43.354 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:17:43.354 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:43.354 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:43.354 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:17:43.354 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:43.355 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:17:43.355 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:43.355 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:17:43.355 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:43.355 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:43.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:43.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:43.356 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:17:43.356 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:43.356 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:43.357 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:17:43.357 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:43.357 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:43.357 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:43.357 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:17:43.358 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:43.358 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:43.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:43.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:43.359 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:17:43.359 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:43.359 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:43.359 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:43.359 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:43.360 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:43.360 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:43.360 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:43.360 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:43.360 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:43.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:43.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:43.361 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:17:43.361 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:43.361 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:43.362 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:43.362 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:43.362 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:43.362 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:43.362 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:43.363 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:43.363 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:43.363 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:43.363 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:17:43.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:43.364 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:17:43.364 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:17:43.364 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:17:43.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:43.364 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:17:43.365 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.952442_679.1 2022-01-11T14:17:48.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:48.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:48.368 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:17:48.413 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:17:48.413 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:17:48.487 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:17:48.488 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:17:48.488 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:17:48.488 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:17:48.489 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:17:48.489 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:17:48.489 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:48.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:48.489 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:17:48.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:48.490 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:17:48.490 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:48.490 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:48.491 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.204054_696.1", 2022-01-11T14:17:48.491 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:48.491 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:17:48.492 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:48.492 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.204054Z", 2022-01-11T14:17:48.492 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:48.492 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:48.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:48.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:48.493 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:17:48.493 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:48.493 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:48.494 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:17:48.494 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:48.494 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:48.494 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:48.495 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:17:48.495 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:48.495 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:48.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:48.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:48.496 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:17:48.496 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:48.496 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:48.497 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:48.497 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:48.497 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:48.497 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:48.497 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:48.498 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:48.498 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:48.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:48.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:48.499 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:17:48.499 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:48.499 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:48.500 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:48.500 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:48.500 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:48.500 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:48.500 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:48.501 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:48.501 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:48.501 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:48.501 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:17:48.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:48.502 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:17:48.502 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:17:48.503 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:17:48.503 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:48.503 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:17:48.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.204054_696.1 2022-01-11T14:17:53.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:53.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:53.507 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:17:53.613 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:17:53.613 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:17:53.906 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:17:53.907 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:17:53.907 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:17:53.907 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:17:53.907 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:17:53.908 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:17:53.908 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:53.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:53.908 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:17:53.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:53.909 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:17:53.909 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:53.909 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:53.909 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.226874_698.1", 2022-01-11T14:17:53.910 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:53.910 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:17:53.910 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:53.910 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.226874Z", 2022-01-11T14:17:53.911 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:53.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:53.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:53.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:53.911 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:17:53.912 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:53.912 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:53.912 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:17:53.912 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:53.913 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:53.913 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:53.913 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:17:53.913 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:53.914 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:53.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:53.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:53.915 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:17:53.915 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:53.915 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:53.915 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:53.915 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:53.916 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:53.916 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:53.916 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:53.916 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:53.917 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:53.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:53.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:53.917 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:17:53.917 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:53.918 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:53.918 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:53.918 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:53.918 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:53.919 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:53.919 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:53.919 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:53.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:53.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:53.920 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:17:53.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:53.920 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:17:53.920 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:17:53.921 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:17:53.921 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:53.921 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:17:53.922 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910389.226874_698.1 2022-01-11T14:17:58.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:58.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:17:58.926 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:17:58.979 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:17:58.980 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:17:59.111 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:17:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:17:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:17:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:17:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:17:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:17:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:17:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:17:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.969581_682.1", 2022-01-11T14:17:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:17:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.969581Z", 2022-01-11T14:17:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:17:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:17:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:17:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:17:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:17:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:17:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:17:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:17:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:17:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:17:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:17:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:17:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:17:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:17:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:17:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:17:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:17:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:17:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:17:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:17:59.126 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:17:59.128 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.969581_682.1 2022-01-11T14:18:04.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:04.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:04.130 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:18:04.174 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:18:04.174 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:18:04.247 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:18:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:18:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:18:04.248 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:18:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:18:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:18:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:04.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:18:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:18:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:04.250 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:18:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:18:04.251 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:18:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:04.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:18:04.253 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:18:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:04.254 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:18:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:04.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:18:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:04.256 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:04.257 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:04.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:18:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:04.259 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:04.260 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:18:04.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:18:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:18:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:18:04.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:04.263 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:18:04.264 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.952442_679.1 2022-01-11T14:18:09.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:09.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:09.267 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:18:09.314 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:18:09.314 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:18:09.386 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:18:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:18:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:18:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:18:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:18:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:18:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:18:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:18:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:18:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:18:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:18:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:18:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:18:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:18:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:18:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:18:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:18:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:18:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:18:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:18:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:09.401 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:18:09.402 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.952442_679.1 2022-01-11T14:18:14.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:14.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:14.405 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:18:14.475 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:18:14.476 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:18:14.824 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:18:14.824 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:18:14.825 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:18:14.825 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:18:14.825 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:18:14.825 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:18:14.825 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:14.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:14.826 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:18:14.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:14.826 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:18:14.827 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:14.827 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:14.827 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.968115_681.1", 2022-01-11T14:18:14.827 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:14.827 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:18:14.828 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:14.828 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.968115Z", 2022-01-11T14:18:14.828 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:14.828 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:14.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:14.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:14.829 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:18:14.829 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:14.829 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:14.829 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:18:14.830 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:14.830 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:14.830 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:14.830 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:18:14.831 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:14.831 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:14.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:14.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:14.831 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:18:14.832 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:14.832 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:14.832 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:14.832 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:14.833 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:14.833 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:14.833 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:14.833 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:14.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:14.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:14.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:14.834 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:18:14.834 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:14.834 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:14.835 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:14.835 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:14.835 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:14.835 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:14.835 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:14.836 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:14.836 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:14.836 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:14.836 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:18:14.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:14.837 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:18:14.837 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:18:14.837 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:18:14.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:14.837 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:18:14.839 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.968115_681.1 2022-01-11T14:18:19.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:19.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:19.842 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:18:19.911 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:18:19.911 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:18:20.073 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:18:20.073 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:18:20.073 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:18:20.074 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:18:20.074 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:18:20.074 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:18:20.074 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:20.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:20.075 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:18:20.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:20.075 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:18:20.075 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:20.076 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:20.076 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.973179_683.1", 2022-01-11T14:18:20.076 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:20.076 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:18:20.077 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:20.077 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.973179Z", 2022-01-11T14:18:20.077 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:20.077 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:20.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:20.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:20.078 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:18:20.078 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:20.079 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:20.079 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:18:20.079 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:20.082 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:20.082 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:20.082 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:18:20.082 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:20.083 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:20.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:20.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:20.083 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:18:20.083 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:20.084 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:20.084 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:20.084 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:20.084 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:20.085 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:20.085 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:20.085 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:20.085 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:20.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:20.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:20.086 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:18:20.086 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:20.087 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:20.087 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:20.087 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:20.087 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:20.087 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:20.088 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:20.088 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:20.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:20.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:20.089 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:18:20.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:20.089 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:18:20.089 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:18:20.089 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:18:20.090 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:20.090 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:18:20.091 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910391.991941_746.1 target=1_1641910388.973179_683.1 2022-01-11T14:18:25.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2022-01-11T14:18:25.094 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-39 2022-01-11T14:18:25.200 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-40 2022-01-11T14:18:25.332 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-41 2022-01-11T14:18:25.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:25.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:25.417 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:18:25.478 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:18:25.479 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:18:25.557 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:18:25.571 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:18:25.571 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:18:25.571 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910705.391393_766.1", 2022-01-11T14:18:25.571 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:18:25.391393Z" 2022-01-11T14:18:25.572 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:18:25.572 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:18:25.572 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:18:25.572 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:18:25.572 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:18:25.573 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:18:25.573 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:18:25.573 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:18:25.573 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:18:25.573 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:18:25.574 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:18:25.574 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:18:25.574 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:18:25.574 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:18:25.575 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:18:25.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:25.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:25.577 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:18:25.645 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:18:25.646 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:18:26.158 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:18:26.159 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:18:26.159 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:18:26.159 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:18:26.159 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:18:26.160 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:18:26.160 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:26.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:26.160 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:18:26.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:26.161 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:18:26.161 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:26.161 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:26.161 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:18:26.162 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:26.162 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:18:26.162 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:26.162 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:18:26.163 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:26.163 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:26.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:26.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:26.163 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:18:26.164 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:26.164 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:26.164 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:18:26.164 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:26.165 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:26.165 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:26.165 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:18:26.165 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:26.166 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:26.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:26.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:26.167 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:18:26.167 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:26.167 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:26.167 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:26.168 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:26.168 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:26.168 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:26.169 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:26.169 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:26.169 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:26.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:26.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:26.170 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:18:26.170 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:26.171 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:26.171 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:26.171 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:26.172 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:26.172 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:26.172 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:26.172 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:26.173 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:26.173 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:26.173 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:18:26.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:26.174 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:18:26.174 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:18:26.174 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:18:26.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:26.175 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:18:26.175 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.952442_679.1 2022-01-11T14:18:30.561 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:18:30.562+0000 7f3d897c4700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2022-01-11T14:18:30.608 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:18:30.608+0000 7f3d897c4700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2022-01-11T14:18:31.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:31.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:31.178 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:18:31.225 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:18:31.226 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:18:31.320 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:18:31.320 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:18:31.321 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:18:31.321 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:18:31.321 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:18:31.321 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:18:31.322 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:31.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:31.322 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:18:31.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:31.323 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:18:31.323 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:31.323 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:31.323 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.075268_690.1", 2022-01-11T14:18:31.324 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:31.324 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:18:31.324 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:31.324 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.075268Z", 2022-01-11T14:18:31.325 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:31.325 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:31.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:31.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:31.325 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:18:31.326 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:31.326 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:31.326 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:18:31.326 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:31.327 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:31.327 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:31.327 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:18:31.327 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:31.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:31.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:31.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:31.328 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:18:31.328 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:31.329 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:31.329 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:31.329 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:31.329 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:31.329 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:31.330 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:31.330 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:31.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:31.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:31.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:31.331 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:18:31.331 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:31.331 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:31.331 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:31.332 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:31.332 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:31.332 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:31.332 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:31.333 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:31.333 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:31.333 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:31.333 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:18:31.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:31.334 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:18:31.334 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:18:31.334 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:18:31.334 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:31.335 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:18:31.336 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.075268_690.1 2022-01-11T14:18:36.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:36.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:36.339 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:18:36.426 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:18:36.427 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:18:36.660 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:18:36.660 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:18:36.660 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:18:36.661 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:18:36.661 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:18:36.661 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:18:36.661 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:36.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:36.662 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:18:36.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:36.662 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:18:36.662 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:36.663 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:36.663 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.230925_700.1", 2022-01-11T14:18:36.663 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:36.663 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:18:36.663 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:36.664 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.230925Z", 2022-01-11T14:18:36.664 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:36.664 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:36.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:36.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:36.665 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:18:36.665 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:36.665 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:36.665 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:18:36.666 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:36.666 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:36.666 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:36.666 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:18:36.666 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:36.667 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:36.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:36.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:36.667 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:18:36.667 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:36.668 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:36.668 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:36.668 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:36.668 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:36.669 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:36.669 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:36.669 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:36.669 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:36.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:36.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:36.670 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:18:36.670 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:36.670 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:36.670 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:36.671 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:36.671 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:36.671 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:36.671 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:36.672 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:36.672 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:36.672 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:36.673 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:18:36.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:36.673 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:18:36.673 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:18:36.674 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:18:36.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:36.674 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:18:36.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.230925_700.1 2022-01-11T14:18:41.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:41.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:41.678 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:18:41.720 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:18:41.721 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:18:41.786 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:18:41.786 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:18:41.786 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:18:41.787 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:18:41.787 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:18:41.787 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:18:41.788 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:41.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:41.788 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:18:41.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:41.788 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:18:41.789 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:41.789 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:41.789 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.198209_694.1", 2022-01-11T14:18:41.789 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:41.789 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:18:41.790 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:41.790 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.198209Z", 2022-01-11T14:18:41.790 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:41.790 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:41.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:41.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:41.791 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:18:41.791 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:41.791 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:41.792 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:18:41.792 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:41.792 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:41.792 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:41.792 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:18:41.793 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:41.793 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:41.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:41.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:41.794 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:18:41.794 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:41.794 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:41.794 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:41.794 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:41.795 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:41.795 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:41.795 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:41.795 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:41.795 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:41.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:41.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:41.796 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:18:41.796 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:41.797 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:41.797 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:41.797 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:41.797 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:41.798 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:41.798 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:41.798 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:41.798 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:41.798 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:41.799 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:18:41.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:41.799 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:18:41.799 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:18:41.799 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:18:41.800 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:41.800 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:18:41.801 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.198209_694.1 2022-01-11T14:18:46.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:46.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:46.804 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:18:46.850 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:18:46.850 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:18:46.920 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:18:46.920 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:18:46.920 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:18:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:18:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:18:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:18:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:46.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:18:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:18:46.922 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.075268_690.1", 2022-01-11T14:18:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:46.923 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:18:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.075268Z", 2022-01-11T14:18:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:46.924 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:18:46.925 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:18:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:46.926 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:18:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:46.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:18:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:46.928 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:46.929 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:46.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:18:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:46.931 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:46.932 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:46.933 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:18:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:18:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:18:46.934 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:18:46.935 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:46.935 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:18:46.936 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.075268_690.1 2022-01-11T14:18:51.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:51.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:51.939 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:18:51.989 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:18:51.989 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:18:52.065 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:18:52.065 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:18:52.066 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:18:52.066 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:18:52.067 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:18:52.067 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:18:52.067 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:52.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:52.068 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:18:52.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:52.069 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:18:52.070 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:52.070 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:52.071 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:18:52.071 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:52.071 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:18:52.072 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:52.072 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:18:52.073 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:52.073 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:52.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:52.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:52.075 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:18:52.075 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:52.076 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:52.076 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:18:52.076 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:52.077 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:52.077 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:52.078 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:18:52.078 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:52.079 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:52.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:52.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:52.080 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:18:52.081 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:52.081 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:52.081 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:52.082 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:52.082 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:52.083 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:52.083 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:52.084 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:52.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:52.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:52.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:52.086 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:18:52.086 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:52.087 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:52.087 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:52.087 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:52.088 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:52.088 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:52.089 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:52.089 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:52.090 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:52.090 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:52.091 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:18:52.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:52.092 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:18:52.092 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:18:52.093 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:18:52.093 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:52.093 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:18:52.096 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.952442_679.1 2022-01-11T14:18:57.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:57.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:18:57.099 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:18:57.147 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:18:57.147 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:18:57.225 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:18:57.226 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:18:57.226 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:18:57.226 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:18:57.226 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:18:57.226 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:18:57.227 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:57.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:57.227 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:18:57.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:57.228 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:18:57.228 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:57.228 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:57.228 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.075268_690.1", 2022-01-11T14:18:57.228 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:57.229 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:18:57.229 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:57.229 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.075268Z", 2022-01-11T14:18:57.229 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:57.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:57.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:57.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:57.230 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:18:57.230 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:57.230 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:57.231 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:18:57.231 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:57.231 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:57.231 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:57.232 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:18:57.232 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:57.232 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:57.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:57.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:57.233 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:18:57.233 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:57.233 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:57.233 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:57.234 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:57.234 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:57.234 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:57.234 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:57.234 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:18:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:18:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:18:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:18:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:18:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:18:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:18:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:18:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:18:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:18:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:18:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:18:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:18:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:18:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:18:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:18:57.239 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:18:57.240 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.075268_690.1 2022-01-11T14:19:02.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:02.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:02.242 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:19:02.316 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:19:02.317 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:19:02.668 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:19:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:19:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:19:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:19:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:19:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:19:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:19:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:19:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:19:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:19:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:19:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:19:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:19:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:19:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:19:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:19:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:02.680 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:02.681 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:19:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:19:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:19:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:19:02.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:02.684 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:19:02.685 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.952442_679.1 2022-01-11T14:19:07.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:07.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:07.688 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:19:07.759 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:19:07.759 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:19:08.043 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:19:08.044 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:19:08.044 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:19:08.044 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:19:08.045 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:19:08.045 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:19:08.045 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:08.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:08.046 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:19:08.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:08.047 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:19:08.047 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:08.047 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:08.047 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.998061_684.1", 2022-01-11T14:19:08.048 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:08.048 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:19:08.048 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:08.049 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.998061Z", 2022-01-11T14:19:08.049 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:08.049 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:08.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:08.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:08.050 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:19:08.050 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:08.050 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:08.051 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:19:08.051 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:08.051 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:08.052 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:08.052 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:19:08.052 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:08.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:08.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:08.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:08.053 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:19:08.054 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:08.054 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:08.054 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:08.054 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:08.055 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:08.055 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:08.055 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:08.055 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:08.056 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:08.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:08.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:08.056 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:19:08.057 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:08.057 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:08.057 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:08.057 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:08.058 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:08.058 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:08.058 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:08.058 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:08.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:08.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:08.059 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:19:08.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:08.060 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:19:08.060 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:19:08.060 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:19:08.060 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:08.061 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:19:08.062 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.998061_684.1 2022-01-11T14:19:13.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:13.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:13.064 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:19:13.108 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:19:13.108 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:19:13.162 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:19:13.162 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:19:13.163 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:19:13.163 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:19:13.163 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:19:13.164 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:19:13.164 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:13.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:13.164 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:19:13.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:13.165 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:19:13.165 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:13.165 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:13.166 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.968115_681.1", 2022-01-11T14:19:13.166 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:13.166 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:19:13.166 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:13.167 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.968115Z", 2022-01-11T14:19:13.167 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:13.167 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:13.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:13.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:13.168 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:19:13.168 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:13.168 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:13.168 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:19:13.169 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:13.169 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:13.169 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:13.169 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:19:13.170 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:13.170 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:13.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:13.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:13.170 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:19:13.171 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:13.171 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:13.171 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:13.171 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:13.172 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:13.172 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:13.172 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:13.172 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:13.172 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:13.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:13.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:13.173 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:19:13.173 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:13.174 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:13.174 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:13.174 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:13.174 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:13.174 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:13.175 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:13.175 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:13.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:13.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:19:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:19:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:19:13.176 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:19:13.177 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:13.177 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:19:13.178 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.968115_681.1 2022-01-11T14:19:18.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:18.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:18.181 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:19:18.226 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:19:18.226 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:19:18.293 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:19:18.293 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:19:18.294 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:19:18.295 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:19:18.295 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:19:18.296 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:19:18.296 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:18.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:18.296 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:19:18.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:18.297 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:19:18.297 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:18.298 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:18.298 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.002077_686.1", 2022-01-11T14:19:18.298 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:18.298 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:19:18.299 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:18.299 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.002077Z", 2022-01-11T14:19:18.299 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:18.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:18.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:18.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:18.300 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:19:18.300 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:18.301 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:18.301 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:19:18.302 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:18.302 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:18.303 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:18.303 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:19:18.303 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:18.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:18.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:18.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:18.305 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:19:18.305 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:18.306 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:18.306 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:18.307 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:18.307 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:18.307 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:18.308 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:18.308 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:18.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:18.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:18.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:18.310 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:19:18.310 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:18.311 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:18.311 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:18.312 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:18.313 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:18.313 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:18.313 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:18.314 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:18.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:18.315 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:18.315 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:19:18.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:18.316 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:19:18.316 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:19:18.317 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:19:18.317 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:18.317 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:19:18.318 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.002077_686.1 2022-01-11T14:19:23.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:23.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:23.321 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:19:23.368 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:19:23.368 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:19:23.439 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:19:23.439 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:19:23.439 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:19:23.440 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:19:23.440 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:19:23.440 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:19:23.440 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:23.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:23.441 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:19:23.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:23.441 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:19:23.442 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:23.442 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:23.442 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.020042_688.1", 2022-01-11T14:19:23.442 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:23.443 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:19:23.443 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:23.443 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.020042Z", 2022-01-11T14:19:23.443 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:23.444 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:23.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:23.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:23.444 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:19:23.444 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:23.445 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:23.445 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:19:23.445 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:23.445 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:23.446 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:23.446 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:19:23.446 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:23.446 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:23.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:23.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:23.447 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:19:23.447 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:23.448 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:23.448 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:23.448 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:23.448 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:23.448 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:23.449 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:23.449 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:23.449 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:23.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:23.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:23.450 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:19:23.450 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:23.450 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:23.451 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:23.451 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:23.451 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:23.451 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:23.452 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:23.452 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:23.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:23.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:23.453 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:19:23.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:23.453 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:19:23.453 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:19:23.454 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:19:23.454 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:23.454 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:19:23.455 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.020042_688.1 2022-01-11T14:19:28.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:28.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:28.458 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:19:28.502 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:19:28.503 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:19:28.574 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:19:28.574 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:19:28.574 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:19:28.575 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:19:28.575 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:19:28.575 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:19:28.575 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:28.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:28.576 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:19:28.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:28.576 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:19:28.576 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:28.577 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:28.577 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.234932_701.1", 2022-01-11T14:19:28.577 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:28.577 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:19:28.578 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:28.578 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.234932Z", 2022-01-11T14:19:28.578 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:28.578 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:28.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:28.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:28.579 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:19:28.579 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:28.580 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:28.580 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:19:28.580 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:28.580 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:28.580 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:28.581 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:19:28.581 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:28.581 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:28.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:28.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:28.582 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:19:28.582 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:28.582 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:28.582 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:28.583 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:28.583 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:28.583 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:28.583 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:28.583 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:28.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:28.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:28.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:28.584 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:19:28.584 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:28.585 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:28.585 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:28.585 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:28.585 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:28.585 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:28.586 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:28.586 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:28.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:28.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:28.586 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:19:28.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:28.587 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:19:28.587 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:19:28.587 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:19:28.587 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:28.588 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:19:28.589 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.234932_701.1 2022-01-11T14:19:33.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:33.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:33.592 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:19:33.641 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:19:33.641 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:19:33.716 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:19:33.717 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:19:33.717 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:19:33.717 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:19:33.717 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:19:33.718 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:19:33.718 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:33.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:33.718 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:19:33.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:33.719 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:19:33.719 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:33.719 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:33.720 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.024470_689.1", 2022-01-11T14:19:33.720 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:33.720 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:19:33.720 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:33.721 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.024470Z", 2022-01-11T14:19:33.721 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:33.721 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:33.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:33.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:33.722 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:19:33.722 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:33.722 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:33.722 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:19:33.723 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:33.723 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:33.723 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:33.723 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:19:33.724 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:33.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:33.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:33.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:33.724 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:19:33.725 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:33.725 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:33.725 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:33.725 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:33.726 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:33.726 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:33.726 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:33.726 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:33.726 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:19:33.727 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:33.728 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:33.729 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:19:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:19:33.730 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:19:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:19:33.731 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:33.731 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:19:33.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.024470_689.1 2022-01-11T14:19:38.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:38.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:38.735 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:19:38.782 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:19:38.782 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:19:38.923 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:19:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:19:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:19:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:19:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:19:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:19:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:19:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:19:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:19:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:19:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:19:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:19:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:19:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:19:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:19:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:19:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:19:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:19:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:19:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:19:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:38.938 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:19:38.938 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.965074_680.1 2022-01-11T14:19:43.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:43.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:43.941 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:19:44.043 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:19:44.043 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:19:44.319 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:19:44.320 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:19:44.320 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:19:44.320 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:19:44.321 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:19:44.321 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:19:44.321 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:44.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:44.322 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:19:44.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:44.322 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:19:44.322 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:44.322 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:44.323 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.204054_696.1", 2022-01-11T14:19:44.323 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:44.323 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:19:44.323 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:44.324 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.204054Z", 2022-01-11T14:19:44.324 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:44.324 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:44.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:44.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:44.325 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:19:44.325 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:44.325 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:44.325 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:19:44.326 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:44.326 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:44.326 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:44.326 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:19:44.327 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:44.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:44.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:44.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:44.328 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:19:44.328 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:44.328 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:44.328 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:44.329 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:44.329 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:44.329 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:44.329 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:44.330 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:44.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:44.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:44.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:44.331 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:19:44.331 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:44.331 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:44.331 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:44.332 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:44.332 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:44.332 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:44.333 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:44.333 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:44.333 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:44.333 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:44.334 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:19:44.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:44.334 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:19:44.334 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:19:44.334 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:19:44.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:44.335 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:19:44.336 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.204054_696.1 2022-01-11T14:19:49.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:49.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:49.339 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:19:49.390 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:19:49.391 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:19:49.567 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:19:49.568 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:19:49.568 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:19:49.568 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:19:49.569 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:19:49.569 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:19:49.569 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:49.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:49.569 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:19:49.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:49.570 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:19:49.570 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:49.570 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:49.571 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:19:49.571 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:49.571 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:19:49.571 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:49.572 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:19:49.572 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:49.572 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:49.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:49.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:49.573 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:19:49.573 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:49.573 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:49.574 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:19:49.574 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:49.574 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:49.574 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:49.574 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:19:49.575 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:49.575 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:49.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:49.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:49.576 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:19:49.576 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:49.576 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:49.576 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:49.576 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:49.577 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:49.577 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:49.577 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:49.577 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:49.578 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:49.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:49.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:49.578 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:19:49.578 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:49.579 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:49.579 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:49.579 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:49.579 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:49.580 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:49.580 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:49.580 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:49.580 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:49.581 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:49.581 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:19:49.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:49.581 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:19:49.581 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:19:49.582 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:19:49.582 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:49.582 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:19:49.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.258675_705.1 2022-01-11T14:19:54.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:54.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:54.586 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:19:54.656 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:19:54.656 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:19:54.783 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:19:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:19:54.784 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:19:54.784 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:19:54.784 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:19:54.784 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:19:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:19:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:19:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.024470_689.1", 2022-01-11T14:19:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:19:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.024470Z", 2022-01-11T14:19:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:19:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:19:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:19:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:19:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:19:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:19:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:19:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:19:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:19:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:54.797 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:19:54.799 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.024470_689.1 2022-01-11T14:19:57.258 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:19:57.254+0000 7fc1e58d4700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2022-01-11T14:19:59.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:59.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:19:59.801 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:19:59.844 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:19:59.845 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:19:59.928 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:19:59.929 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:19:59.929 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:19:59.929 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:19:59.929 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:19:59.930 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:19:59.930 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:59.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:59.930 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:19:59.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:59.931 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:19:59.931 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:59.931 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:59.931 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.002077_686.1", 2022-01-11T14:19:59.932 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:59.932 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:19:59.932 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:59.932 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.002077Z", 2022-01-11T14:19:59.932 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:59.933 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:59.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:59.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:59.933 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:19:59.933 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:59.934 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:59.934 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:19:59.934 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:59.934 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:59.935 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:59.935 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:19:59.935 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:59.935 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:59.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:59.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:59.936 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:19:59.936 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:59.937 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:59.937 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:59.937 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:59.938 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:59.938 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:59.938 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:59.938 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:59.938 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:59.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:59.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:19:59.939 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:19:59.939 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:19:59.940 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:19:59.940 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:19:59.940 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:19:59.940 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:19:59.940 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:19:59.941 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:19:59.941 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:19:59.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:59.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:59.941 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:19:59.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:19:59.942 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:19:59.942 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:19:59.942 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:19:59.943 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:19:59.943 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:19:59.944 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.002077_686.1 2022-01-11T14:20:04.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:04.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:04.947 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:20:04.995 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:20:04.995 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:20:05.049 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:20:05.050 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:20:05.050 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:20:05.051 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:20:05.051 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:20:05.052 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:20:05.052 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:05.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:05.053 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:20:05.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:05.054 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:20:05.054 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:05.054 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:05.055 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.024470_689.1", 2022-01-11T14:20:05.055 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:05.055 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:20:05.055 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:05.056 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.024470Z", 2022-01-11T14:20:05.056 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:05.056 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:05.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:05.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:05.057 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:20:05.057 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:05.057 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:05.057 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:20:05.058 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:05.058 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:05.058 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:05.058 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:20:05.059 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:05.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:05.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:05.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:05.060 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:20:05.060 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:05.060 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:05.060 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:05.060 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:05.061 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:05.061 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:05.061 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:05.061 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:05.062 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:05.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:05.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:05.062 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:20:05.062 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:05.063 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:05.063 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:05.063 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:05.063 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:05.064 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:05.064 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:05.064 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:05.064 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:05.065 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:05.065 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:20:05.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:05.065 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:20:05.065 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:20:05.066 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:20:05.066 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:05.066 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:20:05.067 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.024470_689.1 2022-01-11T14:20:10.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:10.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:10.070 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:20:10.120 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:20:10.120 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:20:10.197 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:20:10.197 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:20:10.197 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:20:10.197 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:20:10.198 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:20:10.198 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:20:10.198 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:10.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:10.199 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:20:10.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:10.199 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:20:10.199 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:10.200 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:10.200 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.973179_683.1", 2022-01-11T14:20:10.200 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:10.200 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:20:10.200 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:10.201 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.973179Z", 2022-01-11T14:20:10.201 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:10.201 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:10.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:10.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:10.202 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:20:10.202 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:10.202 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:10.202 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:20:10.203 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:10.203 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:10.203 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:10.203 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:20:10.204 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:10.204 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:10.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:10.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:10.205 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:20:10.205 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:10.205 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:10.205 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:10.206 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:10.206 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:10.206 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:10.206 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:10.206 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:10.207 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:10.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:10.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:10.207 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:20:10.208 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:10.208 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:10.208 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:10.208 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:10.209 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:10.209 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:10.209 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:10.209 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:10.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:10.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:10.210 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:20:10.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:10.211 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:20:10.211 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:20:10.211 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:20:10.211 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:10.211 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:20:10.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.973179_683.1 2022-01-11T14:20:15.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:15.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:15.215 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:20:15.262 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:20:15.263 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:20:15.328 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:20:15.329 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:20:15.329 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:20:15.329 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:20:15.329 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:20:15.330 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:20:15.330 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:15.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:15.330 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:20:15.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:15.331 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:20:15.331 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:15.331 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:15.332 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.230925_700.1", 2022-01-11T14:20:15.332 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:15.332 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:20:15.332 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:15.333 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.230925Z", 2022-01-11T14:20:15.333 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:15.334 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:15.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:15.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:15.334 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:20:15.335 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:15.335 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:15.335 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:20:15.336 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:15.336 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:15.336 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:15.336 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:20:15.336 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:15.337 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:15.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:15.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:15.337 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:20:15.338 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:15.338 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:15.338 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:15.339 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:15.339 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:15.339 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:15.339 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:15.340 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:15.340 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:15.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:15.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:15.340 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:20:15.341 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:15.341 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:15.341 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:15.341 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:15.342 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:15.342 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:15.342 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:15.342 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:15.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:15.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:15.343 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:20:15.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:15.344 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:20:15.345 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:20:15.346 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:20:15.346 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:15.347 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:20:15.348 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.230925_700.1 2022-01-11T14:20:20.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:20.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:20.351 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:20:20.422 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:20:20.422 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:20:20.823 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:20:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:20:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:20:20.824 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:20:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:20:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:20:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:20.825 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:20:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:20:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.204054_696.1", 2022-01-11T14:20:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:20:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.204054Z", 2022-01-11T14:20:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:20:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:20:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:20:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:20:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:20.832 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:20.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:20:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:20.834 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:20:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:20:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:20:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:20:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:20.837 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:20:20.838 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.204054_696.1 2022-01-11T14:20:25.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:25.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:25.841 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:20:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:20:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:20:25.943 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:20:25.943 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:20:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:20:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:20:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:20:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:20:25.944 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:20:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:25.945 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:20:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.404090_712.1", 2022-01-11T14:20:25.946 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:20:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.404090Z", 2022-01-11T14:20:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:25.947 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:20:25.948 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:20:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:25.949 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:20:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:25.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:20:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:25.951 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:25.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:20:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:25.954 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:25.955 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:20:25.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:20:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:20:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:20:25.957 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:25.957 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:20:25.959 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.404090_712.1 2022-01-11T14:20:30.960 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:30.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:30.962 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:20:31.040 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:20:31.040 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:20:31.220 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:20:31.221 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:20:31.221 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:20:31.221 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:20:31.221 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:20:31.222 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:20:31.222 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:31.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:31.222 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:20:31.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:31.223 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:20:31.223 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:31.223 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:31.224 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.002077_686.1", 2022-01-11T14:20:31.224 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:31.224 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:20:31.224 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:31.225 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.002077Z", 2022-01-11T14:20:31.225 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:31.225 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:31.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:31.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:31.226 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:20:31.226 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:31.226 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:31.227 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:20:31.227 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:31.227 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:31.227 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:31.228 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:20:31.228 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:31.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:31.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:31.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:31.229 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:20:31.229 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:31.229 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:31.229 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:31.229 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:31.230 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:31.230 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:31.230 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:31.230 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:31.230 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:31.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:31.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:31.231 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:20:31.231 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:31.232 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:31.232 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:31.232 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:31.232 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:31.233 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:31.233 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:31.233 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:31.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:31.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:31.234 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:20:31.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:31.234 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:20:31.234 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:20:31.234 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:20:31.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:31.235 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:20:31.236 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.002077_686.1 2022-01-11T14:20:36.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:36.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:36.239 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:20:36.282 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:20:36.283 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:20:36.352 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:20:36.353 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:20:36.353 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:20:36.353 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:20:36.353 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:20:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:20:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:20:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:20:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:20:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:20:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:20:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:36.358 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:20:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:20:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:20:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:20:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:20:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:20:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:20:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:20:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:20:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:36.367 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:20:36.368 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.965074_680.1 2022-01-11T14:20:41.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:41.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:41.371 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:20:41.416 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:20:41.416 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:20:41.478 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:20:41.478 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:20:41.478 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:20:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:20:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:20:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:20:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:20:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:20:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.973179_683.1", 2022-01-11T14:20:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:20:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.973179Z", 2022-01-11T14:20:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:20:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:20:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:20:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:20:41.486 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:41.486 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:41.486 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:41.486 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:20:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:20:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:20:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:20:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:20:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:41.492 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:20:41.493 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.973179_683.1 2022-01-11T14:20:46.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:46.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:46.496 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:20:46.545 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:20:46.546 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:20:46.625 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:20:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:20:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:20:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:20:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:20:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:20:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:20:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:46.627 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:20:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.973179_683.1", 2022-01-11T14:20:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:46.628 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:20:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.973179Z", 2022-01-11T14:20:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:46.629 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:20:46.630 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:20:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:46.631 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:20:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:46.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:20:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:46.633 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:46.634 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:46.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:20:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:46.636 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:20:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:46.638 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:20:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:20:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:20:46.639 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:46.639 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:20:46.641 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.973179_683.1 2022-01-11T14:20:51.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:51.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:51.643 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:20:51.692 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:20:51.692 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:20:51.754 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:20:51.754 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:20:51.755 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:20:51.755 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:20:51.755 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:20:51.755 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:20:51.756 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:51.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:51.756 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:20:51.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:51.756 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:20:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:20:51.757 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:20:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:20:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:51.758 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:20:51.759 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:20:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:51.760 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:20:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:51.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:20:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:51.762 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:51.763 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:51.764 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:20:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:51.765 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:51.766 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:20:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:51.767 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:20:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:20:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:20:51.768 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:51.768 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:20:51.770 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.952442_679.1 2022-01-11T14:20:56.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:56.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:20:56.773 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:20:56.820 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:20:56.821 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:20:56.893 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:20:56.894 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:20:56.894 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:20:56.894 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:20:56.895 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:20:56.895 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:20:56.895 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:56.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:56.895 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:20:56.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:56.896 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:20:56.896 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:56.896 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:56.897 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.973179_683.1", 2022-01-11T14:20:56.897 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:56.897 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:20:56.897 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:56.897 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.973179Z", 2022-01-11T14:20:56.898 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:56.898 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:56.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:56.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:56.899 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:20:56.899 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:56.899 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:56.899 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:20:56.900 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:56.900 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:56.900 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:56.900 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:20:56.900 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:56.901 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:56.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:56.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:56.901 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:20:56.902 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:56.902 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:56.902 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:56.902 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:56.903 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:56.903 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:56.903 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:56.903 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:56.904 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:56.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:56.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:20:56.904 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:20:56.904 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:20:56.905 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:20:56.905 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:20:56.905 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:20:56.905 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:20:56.906 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:20:56.906 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:20:56.906 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:20:56.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:56.907 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:56.907 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:20:56.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:20:56.907 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:20:56.907 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:20:56.908 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:20:56.908 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:20:56.908 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:20:56.910 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.973179_683.1 2022-01-11T14:21:01.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:01.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:01.912 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:21:01.975 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:21:01.975 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:21:02.305 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:21:02.305 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:21:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:21:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:21:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:21:02.306 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:21:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:21:02.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:21:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:21:02.308 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:21:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:21:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:02.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:21:02.310 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:21:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:02.311 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:21:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:02.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:21:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:02.313 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:02.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:21:02.315 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:02.316 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:21:02.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:21:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:21:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:21:02.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:02.318 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:21:02.319 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.965074_680.1 2022-01-11T14:21:05.356 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:21:05.356+0000 7fc1e3ebe700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2022-01-11T14:21:07.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:07.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:07.321 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:21:07.426 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:21:07.427 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:21:07.535 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:21:07.535 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:21:07.535 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:21:07.535 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:21:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:21:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:21:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:21:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:21:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.968115_681.1", 2022-01-11T14:21:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:21:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.968115Z", 2022-01-11T14:21:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:21:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:21:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:21:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:21:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:21:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:21:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:21:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:21:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:21:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:07.549 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:21:07.550 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.968115_681.1 2022-01-11T14:21:12.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:12.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:12.553 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:21:12.610 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:21:12.610 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:21:12.857 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:21:12.857 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:21:12.857 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:21:12.857 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:21:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:21:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:21:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:21:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:21:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.002077_686.1", 2022-01-11T14:21:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:21:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.002077Z", 2022-01-11T14:21:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:21:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:21:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:21:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:21:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:21:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:21:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:21:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:21:12.875 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:21:12.875 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:12.875 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:21:12.877 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.002077_686.1 2022-01-11T14:21:17.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:17.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:17.881 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:21:17.925 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:21:17.925 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:21:17.990 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:21:17.990 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:21:17.991 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:21:17.992 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:21:17.992 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:21:17.993 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:21:17.993 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:17.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:17.994 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:21:17.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:17.995 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:21:17.995 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:17.996 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:17.996 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.020042_688.1", 2022-01-11T14:21:17.997 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:17.997 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:21:17.998 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:17.998 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.020042Z", 2022-01-11T14:21:17.999 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:17.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:17.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:18.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:18.001 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:21:18.001 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:18.001 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:18.002 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:21:18.002 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:18.003 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:18.003 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:18.004 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:21:18.004 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:18.004 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:18.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:18.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:18.006 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:21:18.006 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:18.007 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:18.007 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:18.008 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:18.008 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:18.008 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:18.009 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:18.009 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:18.010 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:18.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:18.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:18.011 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:21:18.012 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:18.012 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:18.012 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:18.013 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:18.013 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:18.014 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:18.014 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:18.015 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:18.015 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:18.015 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:18.016 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:21:18.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:18.017 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:21:18.017 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:21:18.017 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:21:18.018 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:18.018 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:21:18.020 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.020042_688.1 2022-01-11T14:21:23.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:23.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:23.024 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:21:23.073 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:21:23.073 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:21:23.145 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:21:23.145 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:21:23.146 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:21:23.146 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:21:23.146 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:21:23.147 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:21:23.147 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:23.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:23.147 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:21:23.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:23.148 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:21:23.148 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:23.148 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:23.148 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.973179_683.1", 2022-01-11T14:21:23.149 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:23.149 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:21:23.149 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:23.149 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.973179Z", 2022-01-11T14:21:23.149 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:23.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:23.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:23.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:23.150 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:21:23.151 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:23.151 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:23.151 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:21:23.151 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:23.151 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:23.152 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:23.152 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:21:23.152 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:23.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:23.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:23.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:23.153 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:21:23.154 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:23.155 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:23.155 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:23.155 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:23.155 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:23.156 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:23.156 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:23.156 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:23.156 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:23.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:23.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:23.157 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:21:23.157 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:23.157 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:23.158 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:23.158 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:23.158 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:23.159 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:23.159 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:23.159 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:23.159 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:23.159 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:21:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:21:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:21:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:21:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:23.161 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:21:23.163 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.973179_683.1 2022-01-11T14:21:28.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:28.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:28.167 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:21:28.215 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:21:28.215 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:21:28.279 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:21:28.279 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:21:28.280 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:21:28.280 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:21:28.280 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:21:28.281 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:21:28.281 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:28.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:28.282 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:21:28.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:28.283 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:21:28.284 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:28.284 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:28.285 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:21:28.285 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:28.286 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:21:28.286 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:28.287 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:21:28.287 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:28.288 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:28.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:28.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:28.289 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:21:28.289 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:28.290 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:28.290 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:21:28.291 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:28.291 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:28.292 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:28.292 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:21:28.293 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:28.293 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:28.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:28.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:28.294 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:21:28.295 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:28.295 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:28.296 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:28.296 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:28.296 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:28.297 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:28.297 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:28.298 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:28.298 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:28.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:28.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:28.300 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:21:28.300 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:28.300 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:28.301 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:28.301 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:28.302 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:28.302 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:28.302 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:28.303 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:28.303 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:28.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:28.304 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:21:28.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:28.305 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:21:28.305 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:21:28.305 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:21:28.306 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:28.306 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:21:28.308 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.965074_680.1 2022-01-11T14:21:33.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:33.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:33.311 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:21:33.364 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:21:33.365 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:21:33.450 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:21:33.451 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:21:33.451 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:21:33.451 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:21:33.452 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:21:33.452 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:21:33.452 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:33.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:33.453 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:21:33.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:33.453 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:21:33.453 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:33.454 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:33.454 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.204054_696.1", 2022-01-11T14:21:33.454 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:33.455 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:21:33.455 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:33.455 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.204054Z", 2022-01-11T14:21:33.455 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:33.455 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:33.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:33.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:33.456 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:21:33.456 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:33.457 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:33.457 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:21:33.458 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:33.458 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:33.458 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:33.458 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:21:33.458 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:33.459 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:33.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:33.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:33.459 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:21:33.460 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:33.460 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:33.460 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:33.461 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:33.461 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:33.461 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:33.461 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:33.462 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:33.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:33.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:33.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:33.463 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:21:33.463 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:33.463 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:33.464 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:33.464 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:33.464 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:33.465 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:33.465 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:33.465 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:33.465 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:33.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:33.466 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:21:33.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:33.466 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:21:33.466 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:21:33.467 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:21:33.468 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:33.468 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:21:33.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.204054_696.1 2022-01-11T14:21:38.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:38.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:38.474 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:21:38.548 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:21:38.549 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:21:38.935 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:21:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:21:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:21:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:21:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:21:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:21:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:21:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:21:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.230925_700.1", 2022-01-11T14:21:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:21:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.230925Z", 2022-01-11T14:21:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:21:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:21:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:21:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:21:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:21:38.947 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:38.947 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:38.947 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:38.947 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:21:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:21:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:21:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:21:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:38.950 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:21:38.952 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.230925_700.1 2022-01-11T14:21:43.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:43.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:43.955 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:21:44.006 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:21:44.006 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:21:44.089 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:21:44.090 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:21:44.090 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:21:44.090 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:21:44.090 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:21:44.091 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:21:44.091 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:44.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:44.091 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:21:44.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:44.092 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:21:44.092 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:44.092 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:44.092 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.204054_696.1", 2022-01-11T14:21:44.093 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:44.093 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:21:44.093 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:44.093 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.204054Z", 2022-01-11T14:21:44.094 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:44.094 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:44.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:44.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:44.094 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:21:44.095 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:44.095 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:44.095 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:21:44.095 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:44.096 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:44.096 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:44.096 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:21:44.096 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:44.097 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:44.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:44.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:44.097 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:21:44.097 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:44.098 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:44.098 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:44.098 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:44.098 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:44.099 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:44.099 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:44.099 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:44.099 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:44.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:44.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:44.100 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:21:44.100 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:44.100 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:44.101 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:44.101 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:44.101 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:44.101 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:44.102 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:44.102 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:44.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:44.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:44.103 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:21:44.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:44.103 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:21:44.103 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:21:44.103 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:21:44.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:44.104 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:21:44.105 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.204054_696.1 2022-01-11T14:21:49.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:49.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:49.107 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:21:49.178 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:21:49.178 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:21:49.349 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:21:49.349 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:21:49.349 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:21:49.350 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:21:49.350 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:21:49.350 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:21:49.350 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:49.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:49.351 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:21:49.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:49.351 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:21:49.352 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:49.352 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:49.352 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.968115_681.1", 2022-01-11T14:21:49.352 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:49.353 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:21:49.353 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:49.353 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.968115Z", 2022-01-11T14:21:49.353 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:49.353 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:49.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:49.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:49.354 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:21:49.354 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:49.355 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:49.355 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:21:49.355 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:49.355 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:49.355 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:49.356 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:21:49.356 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:49.356 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:49.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:49.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:49.357 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:21:49.357 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:49.357 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:49.357 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:49.358 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:49.358 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:49.358 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:49.358 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:49.359 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:49.359 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:49.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:49.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:49.360 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:21:49.360 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:49.360 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:49.360 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:49.360 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:49.361 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:49.361 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:49.361 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:49.361 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:49.362 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:49.362 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:49.362 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:21:49.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:49.362 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:21:49.363 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:21:49.363 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:21:49.363 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:49.363 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:21:49.365 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.968115_681.1 2022-01-11T14:21:54.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:54.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:54.367 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:21:54.411 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:21:54.411 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:21:54.482 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:21:54.483 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:21:54.483 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:21:54.483 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:21:54.483 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:21:54.484 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:21:54.484 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:54.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:54.484 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:21:54.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:54.485 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:21:54.485 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:54.485 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:54.485 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.000883_685.1", 2022-01-11T14:21:54.486 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:54.486 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:21:54.486 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:54.487 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.000883Z", 2022-01-11T14:21:54.487 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:54.487 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:54.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:54.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:54.488 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:21:54.489 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:54.489 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:54.489 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:21:54.489 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:54.490 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:54.490 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:54.490 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:21:54.490 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:54.491 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:54.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:54.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:54.491 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:21:54.492 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:54.492 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:54.492 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:54.492 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:54.493 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:54.493 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:54.493 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:54.493 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:54.494 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:54.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:54.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:54.494 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:21:54.494 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:54.495 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:54.495 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:54.495 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:54.495 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:54.496 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:54.496 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:54.496 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:54.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:54.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:54.497 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:21:54.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:54.497 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:21:54.497 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:21:54.498 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:21:54.498 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:54.498 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:21:54.499 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.000883_685.1 2022-01-11T14:21:59.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:59.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:21:59.502 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:21:59.546 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:21:59.546 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:21:59.621 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:21:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:21:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:21:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:21:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:21:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:21:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:21:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:21:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.004662_687.1", 2022-01-11T14:21:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:21:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.004662Z", 2022-01-11T14:21:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:21:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:21:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:21:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:21:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:21:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:21:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:21:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:21:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:21:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:21:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:21:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:21:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:21:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:21:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:21:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:21:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:21:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:21:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:21:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:21:59.636 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:21:59.637 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.004662_687.1 2022-01-11T14:22:04.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:04.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:04.640 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:22:04.684 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:22:04.684 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:22:04.754 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:22:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:22:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:22:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:22:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:22:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:22:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:22:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:22:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:22:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:22:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:22:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:22:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:22:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:22:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:22:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:22:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:22:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:22:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:22:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:22:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:04.772 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:22:04.773 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.952442_679.1 2022-01-11T14:22:09.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:09.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:09.776 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:22:09.822 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:22:09.823 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:22:09.890 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:22:09.890 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:22:09.890 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:22:09.891 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:22:09.891 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:22:09.891 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:22:09.891 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:09.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:09.892 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:22:09.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:09.893 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:22:09.893 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:09.893 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:09.893 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.998061_684.1", 2022-01-11T14:22:09.894 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:09.894 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:22:09.894 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:09.895 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.998061Z", 2022-01-11T14:22:09.895 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:09.895 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:09.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:09.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:09.896 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:22:09.896 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:09.896 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:09.897 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:22:09.897 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:09.897 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:09.897 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:09.898 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:22:09.898 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:09.898 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:09.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:09.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:09.899 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:22:09.899 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:09.904 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:09.904 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:09.904 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:09.905 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:09.905 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:09.905 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:09.905 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:09.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:09.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:09.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:09.906 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:22:09.906 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:09.907 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:09.907 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:09.907 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:09.907 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:09.908 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:09.908 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:09.908 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:09.908 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:09.909 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:09.909 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:22:09.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:09.909 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:22:09.910 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:22:09.910 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:22:09.910 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:09.910 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:22:09.912 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.998061_684.1 2022-01-11T14:22:14.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:14.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:14.914 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:22:14.965 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:22:14.965 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:22:15.026 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:22:15.026 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:22:15.027 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:22:15.027 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:22:15.027 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:22:15.028 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:22:15.028 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:15.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:15.028 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:22:15.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:15.029 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:22:15.029 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:15.029 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:15.029 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.230925_700.1", 2022-01-11T14:22:15.030 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:15.030 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:22:15.030 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:15.030 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.230925Z", 2022-01-11T14:22:15.031 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:15.031 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:15.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:15.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:15.032 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:22:15.032 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:15.032 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:15.032 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:22:15.033 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:15.033 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:15.033 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:15.034 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:22:15.034 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:15.034 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:15.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:15.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:15.035 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:22:15.035 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:15.035 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:15.036 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:15.036 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:15.036 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:15.037 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:15.037 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:15.037 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:15.038 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:15.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:15.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:15.038 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:22:15.039 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:15.039 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:15.039 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:15.039 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:15.040 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:15.040 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:15.040 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:15.041 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:15.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:15.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:15.042 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:22:15.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:15.043 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:22:15.043 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:22:15.043 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:22:15.043 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:15.043 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:22:15.045 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.230925_700.1 2022-01-11T14:22:20.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:20.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:20.047 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:22:20.095 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:22:20.095 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:22:20.169 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:22:20.169 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:22:20.169 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:22:20.170 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:22:20.170 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:22:20.170 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:22:20.170 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:20.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:20.171 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:22:20.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:20.171 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:22:20.172 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:20.172 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:20.172 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:22:20.172 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:20.172 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:22:20.173 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:20.173 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:22:20.173 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:20.173 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:20.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:20.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:20.174 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:22:20.174 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:20.175 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:20.175 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:22:20.175 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:20.175 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:20.176 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:20.176 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:22:20.176 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:20.176 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:20.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:20.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:20.177 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:22:20.177 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:20.177 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:20.178 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:20.178 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:20.178 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:20.178 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:20.179 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:20.179 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:20.179 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:20.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:20.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:22:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:20.180 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:20.181 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:20.181 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:20.181 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:20.181 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:20.181 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:20.182 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:20.182 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:20.182 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:22:20.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:20.183 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:22:20.183 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:22:20.183 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:22:20.183 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:20.184 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:22:20.185 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.965074_680.1 2022-01-11T14:22:25.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:25.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:25.188 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:22:25.254 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:22:25.254 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:22:25.662 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:22:25.662 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:22:25.663 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:22:25.663 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:22:25.663 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:22:25.664 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:22:25.664 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:25.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:25.664 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:22:25.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:25.665 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:22:25.665 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:25.665 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:25.666 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.968115_681.1", 2022-01-11T14:22:25.666 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:25.666 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:22:25.666 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:25.666 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.968115Z", 2022-01-11T14:22:25.667 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:25.667 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:25.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:25.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:25.668 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:22:25.668 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:25.668 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:25.668 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:22:25.668 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:25.669 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:25.669 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:25.669 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:22:25.669 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:25.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:25.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:25.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:25.670 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:22:25.671 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:25.671 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:25.671 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:25.671 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:25.672 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:25.672 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:25.672 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:25.672 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:25.673 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:25.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:25.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:25.673 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:22:25.674 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:25.674 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:25.674 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:25.674 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:25.675 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:25.675 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:25.675 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:25.675 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:25.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:25.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:25.676 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:22:25.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:25.676 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:22:25.677 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:22:25.677 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:22:25.677 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:25.677 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:22:25.678 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.968115_681.1 2022-01-11T14:22:30.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:30.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:30.681 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:22:30.730 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:22:30.731 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:22:30.790 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:22:30.791 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:22:30.791 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:22:30.791 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:22:30.792 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:22:30.792 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:22:30.792 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:30.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:30.792 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:22:30.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:30.793 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:22:30.793 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:30.793 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:30.794 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:22:30.794 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:30.794 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:22:30.794 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:30.794 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:22:30.795 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:30.795 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:30.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:30.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:30.796 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:22:30.796 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:30.796 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:30.796 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:22:30.796 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:30.797 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:30.797 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:30.797 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:22:30.797 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:30.798 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:30.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:30.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:30.799 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:22:30.800 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:30.800 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:30.800 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:30.800 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:30.800 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:30.801 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:30.801 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:30.801 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:30.801 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:30.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:30.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:30.802 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:22:30.802 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:30.803 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:30.803 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:30.803 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:30.803 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:30.803 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:30.804 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:30.804 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:30.804 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:30.804 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:30.805 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:22:30.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:30.805 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:22:30.805 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:22:30.805 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:22:30.806 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:30.806 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:22:30.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.952442_679.1 2022-01-11T14:22:35.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:35.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:35.811 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:22:35.873 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:22:35.874 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:22:36.150 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:22:36.150 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:22:36.150 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:22:36.151 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:22:36.151 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:22:36.151 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:22:36.151 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:36.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:36.152 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:22:36.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:36.152 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:22:36.153 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:36.153 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:36.153 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.000883_685.1", 2022-01-11T14:22:36.153 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:36.154 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:22:36.154 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:36.154 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.000883Z", 2022-01-11T14:22:36.154 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:36.155 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:36.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:36.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:36.155 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:22:36.155 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:36.156 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:36.156 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:22:36.156 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:36.156 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:36.157 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:36.157 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:22:36.157 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:36.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:36.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:36.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:36.158 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:22:36.158 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:36.158 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:36.159 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:36.159 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:36.159 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:36.160 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:36.160 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:36.160 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:36.160 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:36.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:36.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:36.161 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:22:36.161 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:36.162 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:36.162 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:36.162 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:36.162 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:36.163 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:36.163 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:36.163 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:36.163 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:36.163 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:36.164 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:22:36.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:36.164 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:22:36.165 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:22:36.165 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:22:36.165 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:36.165 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:22:36.167 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.000883_685.1 2022-01-11T14:22:41.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:41.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:41.170 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:22:41.213 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:22:41.214 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:22:41.283 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:22:41.283 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:22:41.283 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:22:41.284 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:22:41.284 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:22:41.284 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:22:41.284 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:41.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:41.285 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:22:41.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:41.286 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:22:41.286 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:41.286 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:41.286 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:22:41.287 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:41.287 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:22:41.287 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:41.287 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:22:41.288 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:41.288 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:41.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:41.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:41.288 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:22:41.289 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:41.289 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:41.289 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:22:41.290 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:41.290 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:41.290 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:41.290 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:22:41.291 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:41.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:41.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:41.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:41.292 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:22:41.292 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:41.292 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:41.292 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:41.293 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:41.293 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:41.293 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:41.293 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:41.293 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:41.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:41.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:41.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:41.294 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:22:41.295 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:41.295 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:41.295 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:41.295 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:41.296 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:41.296 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:41.296 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:41.296 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:41.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:41.297 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:41.297 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:22:41.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:41.297 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:22:41.298 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:22:41.298 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:22:41.298 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:41.298 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:22:41.300 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.952442_679.1 2022-01-11T14:22:46.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:46.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:46.303 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:22:46.348 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:22:46.348 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:22:46.427 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:22:46.427 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:22:46.428 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:22:46.428 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:22:46.428 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:22:46.428 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:22:46.429 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:46.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:46.429 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:22:46.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:46.430 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:22:46.430 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:46.430 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:46.430 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:22:46.431 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:46.431 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:22:46.431 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:46.431 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:22:46.432 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:46.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:46.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:46.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:46.433 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:22:46.433 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:46.433 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:46.433 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:22:46.434 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:46.434 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:46.434 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:46.434 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:22:46.434 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:46.435 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:46.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:46.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:46.435 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:22:46.436 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:46.436 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:46.436 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:46.436 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:46.437 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:46.437 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:46.438 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:46.438 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:46.438 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:46.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:46.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:46.439 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:22:46.439 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:46.439 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:46.439 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:46.440 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:46.440 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:46.440 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:46.440 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:46.441 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:46.441 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:46.441 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:46.441 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:22:46.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:46.442 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:22:46.442 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:22:46.442 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:22:46.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:46.443 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:22:46.444 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.965074_680.1 2022-01-11T14:22:51.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:51.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:51.447 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:22:51.492 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:22:51.492 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:22:51.552 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:22:51.552 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:22:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:22:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:22:51.553 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:22:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:22:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:51.554 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:22:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:22:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:51.555 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.204054_696.1", 2022-01-11T14:22:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:22:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.204054Z", 2022-01-11T14:22:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:51.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:22:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:51.558 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:22:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:22:51.559 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:51.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:22:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:51.561 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:51.562 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:51.563 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:51.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:22:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:51.564 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:51.565 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:51.566 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:22:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:22:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:22:51.567 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:22:51.568 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:51.568 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:22:51.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.204054_696.1 2022-01-11T14:22:56.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:56.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:22:56.572 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:22:56.623 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:22:56.623 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:22:56.679 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:22:56.680 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:22:56.680 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:22:56.680 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:22:56.680 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:22:56.681 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:22:56.681 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:56.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:56.681 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:22:56.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:56.682 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:22:56.682 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:56.682 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:56.683 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.002077_686.1", 2022-01-11T14:22:56.683 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:56.683 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:22:56.683 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:56.683 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.002077Z", 2022-01-11T14:22:56.684 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:56.684 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:56.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:56.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:56.685 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:22:56.685 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:56.685 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:56.685 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:22:56.685 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:56.686 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:56.686 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:56.686 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:22:56.686 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:56.687 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:56.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:56.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:56.687 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:22:56.688 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:56.688 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:56.688 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:56.688 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:56.688 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:56.689 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:56.689 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:56.689 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:56.689 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:56.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:56.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:22:56.690 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:22:56.690 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:22:56.690 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:22:56.691 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:22:56.691 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:22:56.691 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:22:56.691 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:22:56.692 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:22:56.692 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:22:56.692 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:56.692 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:56.693 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:22:56.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:22:56.693 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:22:56.693 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:22:56.693 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:22:56.694 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:22:56.694 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:22:56.695 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.002077_686.1 2022-01-11T14:23:01.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:01.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:01.698 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:23:01.759 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:23:01.759 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:23:02.140 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:23:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:23:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:23:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:23:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:23:02.141 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:23:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:23:02.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:02.143 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:23:02.143 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:02.143 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:02.143 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.998061_684.1", 2022-01-11T14:23:02.144 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:02.144 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:23:02.144 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:02.144 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.998061Z", 2022-01-11T14:23:02.144 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:02.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:02.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:02.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:02.145 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:23:02.145 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:23:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:23:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:02.147 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:23:02.148 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:02.149 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:02.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:23:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:02.151 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:02.152 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:23:02.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:23:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:23:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:23:02.154 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:02.155 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:23:02.155 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.998061_684.1 2022-01-11T14:23:07.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:07.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:07.158 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:23:07.209 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:23:07.209 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:23:07.308 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:23:07.308 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:23:07.308 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:23:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:23:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:23:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:23:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:23:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:23:07.311 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:07.311 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:07.311 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:23:07.311 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:07.312 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:23:07.312 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:07.312 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:23:07.312 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:23:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:23:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:23:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:23:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:23:07.319 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:07.319 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:07.319 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:07.319 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:23:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:23:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:23:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:23:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:07.322 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:23:07.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.952442_679.1 2022-01-11T14:23:12.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:12.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:12.326 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:23:12.392 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:23:12.393 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:23:12.578 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:23:12.579 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:23:12.579 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:23:12.579 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:23:12.580 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:23:12.580 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:23:12.580 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:12.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:12.580 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:23:12.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:12.581 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:23:12.581 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:12.581 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:12.582 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:23:12.582 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:12.582 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:23:12.582 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:12.583 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:23:12.583 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:12.583 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:12.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:12.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:12.584 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:23:12.584 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:12.584 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:12.584 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:23:12.585 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:12.585 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:12.585 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:12.585 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:23:12.585 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:12.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:12.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:12.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:12.586 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:23:12.587 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:12.587 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:12.587 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:12.587 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:12.587 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:12.588 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:12.588 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:12.588 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:12.588 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:12.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:12.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:12.589 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:23:12.589 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:12.589 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:12.590 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:12.590 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:12.590 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:12.590 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:12.590 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:12.591 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:12.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:12.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:12.591 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:23:12.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:12.592 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:23:12.592 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:23:12.593 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:23:12.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:12.593 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:23:12.594 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.965074_680.1 2022-01-11T14:23:17.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:17.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:17.597 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:23:17.640 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:23:17.641 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:23:17.702 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:23:17.702 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:23:17.703 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:23:17.703 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:23:17.703 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:23:17.703 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:23:17.704 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:17.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:17.704 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:23:17.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:17.705 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:23:17.705 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:17.705 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:17.705 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:23:17.705 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:17.706 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:23:17.706 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:17.706 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:23:17.706 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:17.706 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:17.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:17.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:17.707 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:23:17.707 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:17.708 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:17.708 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:23:17.708 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:17.708 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:17.709 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:17.709 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:23:17.709 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:17.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:17.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:17.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:17.710 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:23:17.710 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:17.710 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:17.710 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:17.711 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:17.711 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:17.711 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:17.711 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:17.711 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:17.712 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:17.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:17.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:17.712 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:23:17.713 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:17.713 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:17.713 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:17.713 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:17.713 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:17.714 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:17.714 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:17.714 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:17.714 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:17.714 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:17.715 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:23:17.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:17.715 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:23:17.715 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:23:17.715 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:23:17.716 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:17.716 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:23:17.717 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.965074_680.1 2022-01-11T14:23:22.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:22.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:22.719 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:23:22.764 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:23:22.764 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:23:22.830 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:23:22.831 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:23:22.831 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:23:22.831 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:23:22.832 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:23:22.832 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:23:22.832 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:22.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:22.833 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:23:22.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:22.833 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:23:22.834 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:22.834 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:22.834 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.968115_681.1", 2022-01-11T14:23:22.834 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:22.835 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:23:22.835 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:22.835 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.968115Z", 2022-01-11T14:23:22.835 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:22.836 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:22.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:22.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:22.836 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:23:22.836 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:22.837 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:22.837 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:23:22.837 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:22.838 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:22.838 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:22.838 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:23:22.839 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:22.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:22.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:22.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:22.840 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:23:22.840 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:22.840 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:22.841 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:22.841 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:22.841 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:22.841 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:22.841 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:22.842 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:22.842 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:22.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:22.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:22.843 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:23:22.844 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:22.844 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:22.844 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:22.844 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:22.844 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:22.845 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:22.845 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:22.845 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:22.845 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:22.846 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:22.846 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:23:22.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:22.846 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:23:22.847 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:23:22.847 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:23:22.847 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:22.847 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:23:22.849 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.968115_681.1 2022-01-11T14:23:27.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:27.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:27.851 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:23:27.897 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:23:27.898 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:23:27.961 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:23:27.962 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:23:27.962 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:23:27.962 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:23:27.963 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:23:27.963 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:23:27.963 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:27.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:27.963 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:23:27.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:27.964 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:23:27.964 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:27.964 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:27.965 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.973179_683.1", 2022-01-11T14:23:27.965 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:27.965 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:23:27.965 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:27.965 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.973179Z", 2022-01-11T14:23:27.966 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:27.966 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:27.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:27.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:27.967 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:23:27.967 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:27.967 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:27.967 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:23:27.968 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:27.968 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:27.968 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:27.968 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:23:27.968 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:27.969 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:27.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:27.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:27.969 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:23:27.970 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:27.970 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:27.970 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:27.970 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:27.971 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:27.971 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:27.971 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:27.971 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:27.971 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:27.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:27.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:27.972 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:23:27.972 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:27.973 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:27.973 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:27.973 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:27.973 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:27.973 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:27.974 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:27.974 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:27.974 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:27.974 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:27.975 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:23:27.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:27.975 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:23:27.975 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:23:27.975 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:23:27.976 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:27.976 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:23:27.977 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910388.973179_683.1 2022-01-11T14:23:32.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:32.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:32.980 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:23:33.029 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:23:33.030 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:23:33.111 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:23:33.112 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:23:33.112 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:23:33.113 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:23:33.113 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:23:33.114 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:23:33.114 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:33.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:33.115 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:23:33.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:33.115 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:23:33.116 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:33.116 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:33.116 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.234932_701.1", 2022-01-11T14:23:33.116 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:33.117 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:23:33.117 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:33.118 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.234932Z", 2022-01-11T14:23:33.118 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:33.118 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:33.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:33.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:33.119 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:23:33.120 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:33.120 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:33.121 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:23:33.121 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:33.122 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:33.122 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:33.122 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:23:33.122 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:33.123 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:33.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:33.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:33.123 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:23:33.124 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:33.124 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:33.124 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:33.124 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:33.125 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:33.125 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:33.125 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:33.125 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:33.125 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:33.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:33.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:33.126 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:23:33.126 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:33.127 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:33.127 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:33.127 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:33.127 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:33.128 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:33.128 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:33.128 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:33.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:33.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:33.129 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:23:33.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:33.129 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:23:33.129 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:23:33.130 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:23:33.130 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:33.130 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:23:33.132 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641910705.391393_766.1 target=1_1641910389.234932_701.1 2022-01-11T14:23:38.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2022-01-11T14:23:38.135 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-42 2022-01-11T14:23:38.154 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-43 2022-01-11T14:23:38.199 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-44 2022-01-11T14:23:38.258 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:23:38.258+0000 7fc1f68f6700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:23:38.259 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2022-01-11T14:23:38.259+0000 7f98c603b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:23:38.259 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:23:38.259+0000 7f3d93bf2700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:23:38.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:38.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:38.389 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:23:38.450 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2022-01-11T14:23:38.450+0000 7f98cb846700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:23:38.622 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:23:38.622 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:23:38.966 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:23:39.038 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:23:39.039 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:23:39.039 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641911018.227948_778.1", 2022-01-11T14:23:39.039 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:23:38.227948Z" 2022-01-11T14:23:39.039 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:23:39.040 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:23:39.040 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:23:39.040 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:23:39.040 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:23:39.040 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:23:39.041 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:23:39.041 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:23:39.041 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:23:39.041 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:23:39.042 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:23:39.042 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:23:39.042 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:23:39.042 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:23:39.043 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:23:39.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:39.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:39.045 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:23:39.094 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:23:39.094 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:23:39.284 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:23:39.284 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:23:39.285 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:23:39.285 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:23:39.285 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:23:39.285 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:23:39.286 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:39.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:39.286 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:23:39.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:39.287 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:23:39.287 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:39.287 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:39.287 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.000883_685.1", 2022-01-11T14:23:39.287 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:39.288 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:23:39.288 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:39.289 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.000883Z", 2022-01-11T14:23:39.289 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:39.289 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:39.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:39.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:39.290 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:23:39.290 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:39.291 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:39.291 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:23:39.291 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:39.291 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:39.291 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:39.292 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:23:39.292 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:39.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:39.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:39.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:39.293 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:23:39.293 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:39.293 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:39.293 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:39.294 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:39.294 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:39.294 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:39.294 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:39.295 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:39.295 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:39.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:39.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:39.295 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:23:39.296 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:39.296 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:39.296 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:39.296 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:39.296 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:39.297 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:39.297 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:39.297 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:39.297 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:39.298 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:39.298 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:23:39.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:39.298 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:23:39.299 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:23:39.299 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:23:39.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:39.299 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:23:39.300 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.000883_685.1 2022-01-11T14:23:39.391 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:23:39.392+0000 7fc1f68f6700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:23:39.663 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:23:39.663+0000 7f3d97bfa700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:23:44.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:44.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:44.303 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:23:44.457 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:23:44.458 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:23:44.586 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:23:44.586 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:23:44.587 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:23:44.587 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:23:44.587 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:23:44.587 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:23:44.588 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:44.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:44.588 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:23:44.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:44.589 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:23:44.589 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:44.589 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:44.589 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.230925_700.1", 2022-01-11T14:23:44.589 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:44.590 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:23:44.590 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:44.590 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.230925Z", 2022-01-11T14:23:44.590 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:44.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:44.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:44.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:44.591 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:23:44.591 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:44.592 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:44.592 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:23:44.592 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:44.592 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:44.593 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:44.593 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:23:44.593 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:44.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:44.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:44.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:44.594 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:23:44.594 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:44.594 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:44.595 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:44.595 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:44.595 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:44.595 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:44.596 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:44.596 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:44.596 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:44.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:44.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:44.597 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:23:44.597 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:44.597 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:44.597 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:44.598 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:44.598 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:44.598 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:44.598 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:44.598 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:44.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:44.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:44.599 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:23:44.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:44.600 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:23:44.600 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:23:44.600 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:23:44.600 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:44.600 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:23:44.602 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.230925_700.1 2022-01-11T14:23:49.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:49.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:49.605 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:23:49.739 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:23:49.740 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:23:49.853 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:23:49.853 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:23:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:23:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:23:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:23:49.854 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:23:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:23:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:23:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:49.856 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.004662_687.1", 2022-01-11T14:23:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:23:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:49.857 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.004662Z", 2022-01-11T14:23:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:23:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:23:49.859 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:23:49.860 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:49.861 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:23:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:49.862 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:49.863 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:23:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:49.865 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:49.866 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:23:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:23:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:23:49.867 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:23:49.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:49.868 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:23:49.869 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.004662_687.1 2022-01-11T14:23:54.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:54.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:23:54.872 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:23:54.964 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:23:54.964 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:23:55.217 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:23:55.217 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:23:55.217 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:23:55.217 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:23:55.218 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:23:55.218 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:23:55.218 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:55.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:55.219 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:23:55.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:55.219 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:23:55.219 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:55.219 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:55.220 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:23:55.220 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:55.220 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:23:55.220 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:55.221 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:23:55.221 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:55.221 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:55.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:55.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:55.222 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:23:55.222 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:55.222 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:55.222 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:23:55.223 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:55.223 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:55.223 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:55.223 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:23:55.223 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:55.224 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:55.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:55.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:55.224 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:23:55.224 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:55.225 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:55.225 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:55.225 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:55.225 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:55.226 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:55.226 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:55.226 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:55.226 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:55.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:55.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:23:55.227 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:23:55.227 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:23:55.227 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:23:55.228 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:23:55.228 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:23:55.228 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:23:55.228 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:23:55.228 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:23:55.229 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:23:55.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:55.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:55.229 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:23:55.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:23:55.230 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:23:55.230 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:23:55.230 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:23:55.230 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:23:55.230 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:23:55.232 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.952442_679.1 2022-01-11T14:24:00.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:00.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:00.236 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:24:00.416 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:24:00.416 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:24:00.500 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:24:00.500 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:24:00.500 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:24:00.501 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:24:00.501 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:24:00.501 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:24:00.501 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:00.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:00.502 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:24:00.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:00.502 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:24:00.503 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:00.503 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:00.503 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.973179_683.1", 2022-01-11T14:24:00.503 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:00.504 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:24:00.504 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:00.504 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.973179Z", 2022-01-11T14:24:00.504 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:00.504 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:00.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:00.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:00.505 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:24:00.505 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:00.505 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:00.506 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:24:00.506 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:00.506 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:00.506 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:00.507 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:24:00.507 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:00.507 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:00.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:00.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:00.508 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:24:00.508 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:00.508 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:00.508 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:00.509 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:00.509 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:00.509 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:00.509 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:00.510 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:00.510 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:00.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:00.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:00.510 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:24:00.511 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:00.511 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:00.511 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:00.511 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:00.512 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:00.512 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:00.512 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:00.512 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:00.512 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:00.513 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:00.513 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:24:00.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:00.513 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:24:00.514 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:24:00.514 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:24:00.514 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:00.514 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:24:00.515 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.973179_683.1 2022-01-11T14:24:05.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:05.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:05.519 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:24:05.617 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:24:05.617 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:24:05.752 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:24:05.753 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:24:05.753 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:24:05.753 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:24:05.753 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:24:05.754 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:24:05.754 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:05.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:05.754 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:24:05.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:05.755 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:24:05.755 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:05.755 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:05.755 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.002077_686.1", 2022-01-11T14:24:05.756 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:05.756 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:24:05.756 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:05.756 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.002077Z", 2022-01-11T14:24:05.757 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:05.757 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:05.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:05.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:05.758 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:24:05.758 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:05.758 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:05.758 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:24:05.759 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:05.759 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:05.759 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:05.760 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:24:05.760 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:05.760 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:05.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:05.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:05.761 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:24:05.761 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:05.761 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:05.761 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:05.762 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:05.762 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:05.762 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:05.762 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:05.763 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:05.763 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:05.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:05.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:05.763 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:24:05.764 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:05.764 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:05.764 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:05.764 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:05.765 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:05.765 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:05.765 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:05.765 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:05.765 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:05.766 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:05.766 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:24:05.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:05.766 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:24:05.767 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:24:05.767 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:24:05.767 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:05.767 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:24:05.769 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.002077_686.1 2022-01-11T14:24:10.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:10.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:10.771 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:24:11.171 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:24:11.172 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:24:11.243 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:24:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:24:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:24:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:24:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:24:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:24:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:24:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:24:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:24:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:24:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:24:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:24:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:24:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:24:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:24:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:11.254 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:24:11.254 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:11.254 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:11.254 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:11.254 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:11.255 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:11.255 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:11.255 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:11.255 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:11.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:11.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:11.256 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:24:11.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:11.256 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:24:11.257 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:24:11.257 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:24:11.257 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:11.257 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:24:11.258 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.965074_680.1 2022-01-11T14:24:16.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:16.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:16.261 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:24:16.304 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:24:16.304 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:24:16.376 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:24:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:24:16.376 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:24:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:24:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:24:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:24:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:16.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:24:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:24:16.378 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.002077_686.1", 2022-01-11T14:24:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:16.379 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:24:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.002077Z", 2022-01-11T14:24:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:16.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:24:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:16.381 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:24:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:16.382 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:24:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:16.383 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:24:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:16.384 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:16.385 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:16.386 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:24:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:16.387 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:16.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:24:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:24:16.389 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:24:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:24:16.390 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:16.390 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:24:16.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.002077_686.1 2022-01-11T14:24:21.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:21.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:21.395 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:24:21.438 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:24:21.439 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:24:21.492 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:24:21.492 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:24:21.492 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:24:21.493 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:24:21.493 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:24:21.493 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:24:21.493 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:21.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:21.494 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:24:21.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:21.494 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:24:21.495 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:21.495 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:21.495 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.230925_700.1", 2022-01-11T14:24:21.495 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:21.495 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:24:21.496 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:21.496 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.230925Z", 2022-01-11T14:24:21.496 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:21.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:21.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:21.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:21.497 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:24:21.497 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:21.497 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:21.498 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:24:21.498 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:21.498 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:21.498 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:21.499 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:24:21.499 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:21.499 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:21.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:21.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:21.500 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:24:21.500 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:21.500 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:21.501 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:21.501 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:21.501 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:21.501 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:21.502 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:21.502 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:21.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:21.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:21.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:21.503 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:24:21.503 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:21.503 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:21.503 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:21.504 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:21.504 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:21.504 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:21.504 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:21.505 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:21.505 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:21.505 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:21.506 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:24:21.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:21.506 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:24:21.506 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:24:21.507 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:24:21.507 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:21.507 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:24:21.508 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.230925_700.1 2022-01-11T14:24:26.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:26.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:26.512 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:24:26.563 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:24:26.564 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:24:26.636 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:24:26.637 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:24:26.637 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:24:26.637 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:24:26.638 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:24:26.638 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:24:26.638 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:26.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:26.639 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:24:26.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:26.639 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:24:26.640 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:26.640 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:26.640 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.134640_692.1", 2022-01-11T14:24:26.641 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:26.641 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:24:26.641 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:26.642 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.134640Z", 2022-01-11T14:24:26.642 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:26.642 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:26.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:26.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:26.643 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:24:26.643 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:26.644 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:26.644 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:24:26.644 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:26.645 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:26.645 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:26.645 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:24:26.646 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:26.646 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:26.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:26.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:26.647 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:24:26.648 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:26.648 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:26.648 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:26.649 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:26.649 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:26.650 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:26.650 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:26.650 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:26.651 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:26.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:26.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:26.652 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:24:26.652 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:26.652 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:26.653 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:26.653 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:26.653 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:26.654 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:26.654 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:26.654 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:26.655 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:26.655 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:26.655 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:24:26.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:26.656 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:24:26.656 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:24:26.656 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:24:26.657 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:26.657 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:24:26.659 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.134640_692.1 2022-01-11T14:24:31.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:31.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:31.663 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:24:31.711 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:24:31.712 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:24:31.778 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:24:31.778 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:24:31.779 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:24:31.779 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:24:31.779 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:24:31.779 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:24:31.780 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:31.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:31.780 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:24:31.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:31.781 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:24:31.781 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:31.781 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:31.781 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.000883_685.1", 2022-01-11T14:24:31.782 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:31.782 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:24:31.782 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:31.782 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.000883Z", 2022-01-11T14:24:31.782 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:31.783 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:31.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:31.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:31.783 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:24:31.784 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:31.784 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:31.784 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:24:31.784 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:31.785 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:31.785 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:31.785 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:24:31.785 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:31.785 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:31.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:31.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:31.786 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:24:31.786 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:31.787 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:31.787 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:31.787 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:31.787 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:31.788 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:31.788 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:31.788 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:31.788 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:31.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:31.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:31.789 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:24:31.789 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:31.789 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:31.790 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:31.790 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:31.790 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:31.790 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:31.790 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:31.791 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:31.791 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:31.791 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:31.791 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:24:31.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:31.792 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:24:31.792 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:24:31.792 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:24:31.793 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:31.793 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:24:31.794 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.000883_685.1 2022-01-11T14:24:36.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:36.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:36.797 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:24:36.841 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:24:36.842 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:24:36.920 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:24:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:24:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:24:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:24:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:24:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:24:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:24:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:24:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.969581_682.1", 2022-01-11T14:24:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:24:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.969581Z", 2022-01-11T14:24:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:24:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:24:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:24:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:24:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:24:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:24:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:24:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:24:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:24:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:36.935 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:24:36.936 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.969581_682.1 2022-01-11T14:24:41.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:41.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:41.939 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:24:41.982 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:24:41.983 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:24:42.060 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:24:42.060 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:24:42.060 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:24:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:24:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:24:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:24:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:24:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:24:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.079571_691.1", 2022-01-11T14:24:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:42.064 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:24:42.064 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:42.064 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.079571Z", 2022-01-11T14:24:42.064 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:42.065 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:42.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:42.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:42.065 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:24:42.065 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:24:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:24:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:24:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:42.069 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:42.069 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:42.069 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:42.069 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:42.070 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:42.070 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:42.070 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:42.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:42.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:42.071 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:24:42.071 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:42.071 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:42.071 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:24:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:42.074 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:24:42.074 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:24:42.074 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:24:42.074 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:42.075 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:24:42.076 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.079571_691.1 2022-01-11T14:24:47.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:47.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:47.080 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:24:47.122 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:24:47.122 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:24:47.177 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:24:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:24:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:24:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:24:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:24:47.179 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:24:47.179 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:47.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:47.179 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:24:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:24:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.079571_691.1", 2022-01-11T14:24:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:24:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.079571Z", 2022-01-11T14:24:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:24:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:24:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:24:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:24:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:47.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:47.190 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:24:47.190 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:47.191 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:47.191 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:24:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:24:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:24:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:24:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:47.197 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:24:47.199 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.079571_691.1 2022-01-11T14:24:52.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:52.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:52.203 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:24:52.260 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:24:52.260 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:24:52.321 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:24:52.322 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:24:52.322 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:24:52.323 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:24:52.323 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:24:52.323 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:24:52.323 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:52.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:52.324 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:24:52.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:52.324 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:24:52.324 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:52.325 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:52.325 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.230925_700.1", 2022-01-11T14:24:52.325 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:52.325 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:24:52.326 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:52.326 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.230925Z", 2022-01-11T14:24:52.326 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:52.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:52.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:52.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:52.327 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:24:52.328 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:52.328 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:52.328 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:24:52.328 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:52.329 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:52.329 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:52.329 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:24:52.330 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:52.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:52.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:52.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:52.331 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:24:52.331 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:52.331 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:52.332 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:52.332 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:52.332 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:52.332 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:52.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:24:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:52.334 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:52.335 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:52.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:24:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:24:52.337 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:24:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:24:52.338 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:52.338 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:24:52.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.230925_700.1 2022-01-11T14:24:57.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:57.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:24:57.343 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:24:57.385 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:24:57.385 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:24:57.454 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:24:57.454 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:24:57.454 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:24:57.455 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:24:57.455 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:24:57.455 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:24:57.455 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:57.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:57.456 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:24:57.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:57.456 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:24:57.457 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:57.457 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:57.457 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.973179_683.1", 2022-01-11T14:24:57.457 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:57.458 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:24:57.458 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:57.458 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.973179Z", 2022-01-11T14:24:57.458 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:57.459 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:57.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:57.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:57.459 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:24:57.459 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:57.460 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:57.460 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:24:57.460 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:57.460 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:57.461 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:57.461 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:24:57.461 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:57.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:57.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:57.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:57.462 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:24:57.462 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:57.463 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:57.463 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:57.463 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:57.463 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:57.463 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:57.464 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:57.464 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:57.464 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:57.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:57.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:24:57.465 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:24:57.465 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:24:57.465 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:24:57.466 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:24:57.466 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:24:57.466 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:24:57.466 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:24:57.467 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:24:57.467 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:24:57.467 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:57.467 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:57.467 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:24:57.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:24:57.468 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:24:57.468 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:24:57.468 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:24:57.469 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:24:57.469 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:24:57.470 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.973179_683.1 2022-01-11T14:25:02.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:02.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:02.474 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:25:02.542 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:25:02.542 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:25:02.663 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:25:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:25:02.663 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:25:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:25:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:25:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:25:02.664 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:25:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:25:02.665 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.198209_694.1", 2022-01-11T14:25:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:02.666 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:25:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.198209Z", 2022-01-11T14:25:02.667 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:02.668 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:25:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:25:02.669 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:25:02.670 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:02.671 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:25:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:02.672 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:02.673 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:02.674 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:25:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:02.675 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:02.676 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:25:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:02.677 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:25:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:25:02.678 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:25:02.679 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:02.679 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:25:02.680 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.198209_694.1 2022-01-11T14:25:07.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:07.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:07.683 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:25:07.750 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:25:07.750 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:25:07.842 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:25:07.842 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:25:07.842 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:25:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:25:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:25:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:25:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:25:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:25:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:25:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:25:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:25:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:07.847 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:25:07.847 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:07.847 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:07.847 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:25:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:25:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:25:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:25:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:25:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:25:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:25:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:25:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:07.857 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:25:07.858 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.952442_679.1 2022-01-11T14:25:12.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:12.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:12.861 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:25:12.915 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:25:12.916 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:25:12.985 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:25:12.986 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:25:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:25:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:25:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:25:12.988 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:25:12.988 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:12.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:12.989 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:25:12.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:12.990 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:25:12.990 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:12.991 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:12.991 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.973179_683.1", 2022-01-11T14:25:12.992 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:12.992 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:25:12.993 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:12.993 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.973179Z", 2022-01-11T14:25:12.993 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:12.994 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:12.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:12.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:12.995 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:25:12.996 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:12.996 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:12.996 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:25:12.997 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:12.997 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:12.998 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:12.998 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:25:12.999 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:12.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:13.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:13.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:13.001 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:25:13.001 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:13.003 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:13.004 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:13.004 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:13.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:25:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:25:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:25:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:25:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:25:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:13.011 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:25:13.012 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.973179_683.1 2022-01-11T14:25:18.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:18.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:18.015 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:25:18.088 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:25:18.088 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:25:18.256 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:25:18.257 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:25:18.257 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:25:18.258 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:25:18.258 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:25:18.259 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:25:18.259 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:18.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:18.260 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:25:18.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:18.261 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:25:18.262 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:18.262 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:18.263 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.020042_688.1", 2022-01-11T14:25:18.263 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:18.264 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:25:18.264 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:18.265 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.020042Z", 2022-01-11T14:25:18.265 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:18.266 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:18.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:18.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:18.267 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:25:18.267 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:18.268 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:18.268 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:25:18.268 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:18.268 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:18.269 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:18.269 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:25:18.269 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:18.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:18.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:18.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:18.270 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:25:18.270 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:18.270 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:18.271 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:18.271 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:18.271 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:18.272 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:18.272 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:18.272 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:18.272 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:18.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:18.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:18.273 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:25:18.273 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:18.274 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:18.274 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:18.274 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:18.274 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:18.275 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:18.275 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:18.275 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:18.275 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:18.275 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:18.276 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:25:18.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:18.276 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:25:18.276 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:25:18.277 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:25:18.277 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:18.277 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:25:18.277 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.020042_688.1 2022-01-11T14:25:23.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:23.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:23.280 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:25:23.489 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:25:23.489 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:25:23.718 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:25:23.719 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:25:23.719 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:25:23.720 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:25:23.720 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:25:23.720 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:25:23.721 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:23.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:23.721 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:25:23.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:23.722 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:25:23.722 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:23.723 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:23.723 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.969581_682.1", 2022-01-11T14:25:23.723 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:23.723 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:25:23.724 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:23.724 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.969581Z", 2022-01-11T14:25:23.724 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:23.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:23.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:23.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:23.725 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:25:23.725 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:23.726 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:23.726 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:25:23.726 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:23.726 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:23.726 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:23.727 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:25:23.727 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:23.727 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:23.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:23.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:23.728 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:25:23.728 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:23.728 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:23.728 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:23.729 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:23.729 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:23.729 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:23.729 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:23.729 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:23.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:23.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:23.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:23.731 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:25:23.731 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:23.731 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:23.731 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:23.731 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:23.732 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:23.732 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:23.732 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:23.732 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:23.732 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:23.733 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:23.733 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:25:23.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:23.733 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:25:23.734 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:25:23.734 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:25:23.734 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:23.734 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:25:23.736 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.969581_682.1 2022-01-11T14:25:28.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:28.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:28.738 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:25:28.804 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:25:28.805 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:25:28.927 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:25:28.927 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:25:28.928 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:25:28.928 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:25:28.928 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:25:28.928 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:25:28.929 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:28.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:28.929 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:25:28.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:28.930 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:25:28.930 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:28.930 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:28.930 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:25:28.930 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:28.931 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:25:28.931 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:28.931 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:25:28.931 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:28.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:28.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:28.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:28.932 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:25:28.932 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:28.933 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:28.933 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:25:28.933 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:28.933 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:28.934 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:28.934 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:25:28.934 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:28.934 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:28.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:28.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:28.935 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:25:28.935 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:28.935 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:28.936 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:28.936 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:28.936 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:28.936 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:28.936 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:28.937 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:28.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:28.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:28.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:28.938 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:25:28.938 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:28.938 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:28.938 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:28.938 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:28.939 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:28.939 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:28.939 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:28.939 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:28.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:28.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:28.940 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:25:28.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:28.940 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:25:28.941 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:25:28.941 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:25:28.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:28.941 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:25:28.942 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.952442_679.1 2022-01-11T14:25:33.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:33.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:33.946 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:25:34.005 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:25:34.005 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:25:34.086 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:25:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:25:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:25:34.087 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:25:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:25:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:25:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:34.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:25:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:25:34.089 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.002077_686.1", 2022-01-11T14:25:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:34.090 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:25:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.002077Z", 2022-01-11T14:25:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:34.091 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:25:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:34.092 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:25:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:34.093 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:25:34.094 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:34.095 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:25:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:34.096 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:25:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:34.099 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:34.101 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:25:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:34.102 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:25:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:25:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:25:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:34.103 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:25:34.105 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.002077_686.1 2022-01-11T14:25:39.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:39.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:39.107 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:25:39.159 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:25:39.159 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:25:39.230 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:25:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:25:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:25:39.231 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:25:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:25:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:25:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:39.232 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:25:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:25:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:39.233 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.968115_681.1", 2022-01-11T14:25:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:25:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:39.234 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.968115Z", 2022-01-11T14:25:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:39.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:25:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:39.236 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:25:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:39.237 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:25:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:39.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:25:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:39.239 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:39.240 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:39.241 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:25:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:39.242 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:25:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:25:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:25:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:25:39.245 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:39.245 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:25:39.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.968115_681.1 2022-01-11T14:25:44.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:44.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:44.249 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:25:44.338 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:25:44.339 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:25:44.549 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:25:44.550 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:25:44.550 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:25:44.550 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:25:44.551 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:25:44.551 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:25:44.551 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:44.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:44.552 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:25:44.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:44.552 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:25:44.552 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:44.552 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:44.553 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.000883_685.1", 2022-01-11T14:25:44.553 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:44.553 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:25:44.553 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:44.554 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.000883Z", 2022-01-11T14:25:44.554 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:44.554 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:44.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:44.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:44.555 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:25:44.555 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:44.555 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:44.555 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:25:44.556 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:44.556 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:44.556 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:44.556 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:25:44.556 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:44.557 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:44.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:44.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:44.557 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:25:44.558 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:44.558 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:44.558 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:44.558 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:44.559 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:44.559 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:44.559 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:44.559 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:44.560 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:44.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:44.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:44.560 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:25:44.560 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:44.561 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:44.561 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:44.561 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:44.561 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:44.561 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:44.562 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:44.562 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:44.562 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:44.562 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:44.563 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:25:44.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:44.563 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:25:44.563 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:25:44.564 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:25:44.564 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:44.564 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:25:44.565 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.000883_685.1 2022-01-11T14:25:49.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:49.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:49.568 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:25:49.653 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:25:49.654 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:25:49.762 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:25:49.762 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:25:49.762 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:25:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:25:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:25:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:25:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:25:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:25:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:25:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:25:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:25:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:25:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:25:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:25:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:25:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:25:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:25:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:25:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:25:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:25:49.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:49.777 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:25:49.778 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.952442_679.1 2022-01-11T14:25:54.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:54.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:25:54.781 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:25:54.830 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:25:54.831 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:25:55.033 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:25:55.033 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:25:55.034 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:25:55.034 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:25:55.034 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:25:55.034 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:25:55.035 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:55.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:55.035 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:25:55.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:55.036 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:25:55.036 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:55.037 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:55.037 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:25:55.037 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:55.037 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:25:55.038 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:55.038 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:25:55.038 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:55.038 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:55.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:55.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:55.039 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:25:55.040 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:55.040 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:55.040 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:25:55.040 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:55.040 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:55.041 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:55.041 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:25:55.041 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:55.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:55.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:55.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:55.042 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:25:55.042 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:55.043 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:55.043 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:55.043 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:55.043 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:55.044 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:55.044 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:55.044 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:55.044 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:55.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:55.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:25:55.045 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:25:55.045 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:25:55.045 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:25:55.046 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:25:55.046 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:25:55.046 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:25:55.046 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:25:55.047 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:25:55.047 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:25:55.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:55.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:55.048 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:25:55.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:25:55.048 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:25:55.048 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:25:55.048 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:25:55.049 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:25:55.049 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:25:55.052 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.965074_680.1 2022-01-11T14:26:00.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:00.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:00.056 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:26:00.141 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:26:00.141 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:26:00.293 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:26:00.293 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:26:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:26:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:26:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:26:00.294 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:26:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:26:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:00.295 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:26:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.138030_693.1", 2022-01-11T14:26:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:00.296 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:26:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.138030Z", 2022-01-11T14:26:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:00.297 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:26:00.298 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:00.299 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:00.299 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:26:00.299 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:00.299 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:00.299 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:00.300 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:26:00.300 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:00.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:00.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:00.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:00.301 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:26:00.301 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:00.301 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:00.301 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:00.302 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:00.302 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:00.302 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:00.302 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:00.303 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:00.303 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:00.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:00.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:00.303 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:26:00.304 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:00.304 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:00.304 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:00.304 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:00.304 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:00.305 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:00.305 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:00.305 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:00.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:00.306 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:00.306 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:26:00.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:00.306 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:26:00.306 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:26:00.307 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:26:00.307 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:00.307 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:26:00.308 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.138030_693.1 2022-01-11T14:26:05.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:05.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:05.311 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:26:05.389 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:26:05.390 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:26:05.625 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:26:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:26:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:26:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:26:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:26:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:26:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:26:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:26:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:05.629 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:05.629 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.075268_690.1", 2022-01-11T14:26:05.629 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:05.629 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:26:05.630 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:05.630 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.075268Z", 2022-01-11T14:26:05.630 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:05.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:05.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:05.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:05.631 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:26:05.631 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:05.631 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:05.632 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:26:05.632 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:05.632 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:05.632 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:05.632 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:26:05.633 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:05.633 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:05.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:05.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:05.634 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:26:05.634 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:05.634 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:05.634 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:05.635 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:05.635 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:05.635 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:05.635 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:05.635 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:05.636 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:05.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:05.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:05.636 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:26:05.637 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:05.637 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:05.637 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:05.637 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:05.637 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:05.638 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:05.638 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:05.638 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:05.638 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:05.639 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:05.639 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:26:05.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:05.639 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:26:05.640 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:26:05.640 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:26:05.640 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:05.640 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:26:05.642 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.075268_690.1 2022-01-11T14:26:10.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:10.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:10.644 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:26:10.711 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:26:10.712 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:26:10.864 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:26:10.864 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:26:10.864 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:26:10.865 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:26:10.865 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:26:10.865 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:26:10.865 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:10.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:10.866 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:26:10.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:10.866 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:26:10.866 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:10.867 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:10.867 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:26:10.867 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:10.867 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:26:10.868 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:10.868 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:26:10.868 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:10.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:10.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:10.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:10.869 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:26:10.869 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:10.869 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:10.870 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:26:10.870 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:10.870 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:10.870 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:10.871 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:26:10.871 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:10.871 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:10.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:10.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:10.872 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:26:10.872 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:10.872 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:10.872 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:10.873 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:10.873 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:10.873 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:10.873 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:10.873 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:10.874 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:10.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:10.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:10.874 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:26:10.875 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:10.875 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:10.875 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:10.875 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:10.876 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:10.876 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:10.876 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:10.876 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:10.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:10.877 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:10.877 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:26:10.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:10.877 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:26:10.878 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:26:10.878 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:26:10.878 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:10.878 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:26:10.880 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.965074_680.1 2022-01-11T14:26:15.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:16.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:16.355 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:26:16.400 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:26:16.400 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:26:16.476 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:26:16.476 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:26:16.476 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:26:16.477 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:26:16.477 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:26:16.477 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:26:16.477 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:16.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:16.478 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:26:16.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:16.478 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:26:16.478 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:16.479 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:16.479 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:26:16.479 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:16.479 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:26:16.479 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:16.480 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:26:16.480 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:16.480 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:16.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:16.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:16.481 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:26:16.481 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:16.481 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:16.482 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:26:16.482 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:16.482 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:16.482 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:16.483 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:26:16.483 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:16.483 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:16.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:16.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:16.484 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:26:16.484 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:16.484 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:16.484 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:16.484 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:16.485 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:16.485 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:16.485 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:16.485 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:16.486 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:16.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:16.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:16.486 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:26:16.487 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:16.487 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:16.487 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:16.487 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:16.487 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:16.488 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:16.488 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:16.488 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:16.488 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:16.489 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:16.489 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:26:16.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:16.489 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:26:16.489 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:26:16.490 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:26:16.490 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:16.490 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:26:16.491 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.965074_680.1 2022-01-11T14:26:21.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:21.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:21.494 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:26:21.538 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:26:21.538 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:26:21.612 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:26:21.613 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:26:21.613 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:26:21.613 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:26:21.613 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:26:21.614 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:26:21.614 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:21.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:21.614 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:26:21.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:21.615 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:26:21.615 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:21.615 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:21.615 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.969581_682.1", 2022-01-11T14:26:21.616 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:21.616 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:26:21.616 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:21.617 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.969581Z", 2022-01-11T14:26:21.617 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:21.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:21.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:21.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:21.618 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:26:21.618 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:21.618 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:21.619 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:26:21.619 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:21.619 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:21.619 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:21.619 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:26:21.620 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:21.620 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:21.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:21.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:21.621 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:26:21.621 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:21.621 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:21.621 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:21.621 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:21.622 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:21.622 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:21.622 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:21.622 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:21.623 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:21.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:21.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:21.623 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:26:21.623 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:21.624 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:21.624 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:21.624 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:21.624 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:21.625 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:21.625 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:21.625 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:21.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:21.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:21.626 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:26:21.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:21.626 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:26:21.626 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:26:21.627 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:26:21.627 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:21.627 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:26:21.628 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.969581_682.1 2022-01-11T14:26:26.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:26.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:26.631 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:26:26.674 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:26:26.675 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:26:26.751 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:26:26.751 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:26:26.752 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:26:26.752 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:26:26.752 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:26:26.752 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:26:26.752 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:26.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:26.753 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:26:26.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:26.753 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:26:26.754 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:26.754 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:26.754 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.075268_690.1", 2022-01-11T14:26:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:26:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.075268Z", 2022-01-11T14:26:26.756 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:26.756 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:26.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:26.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:26.756 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:26:26.757 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:26.757 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:26.757 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:26:26.757 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:26.758 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:26.758 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:26.758 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:26:26.758 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:26.758 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:26.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:26.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:26.759 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:26:26.759 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:26.760 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:26.760 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:26.760 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:26.760 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:26.760 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:26.761 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:26.761 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:26.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:26.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:26.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:26.762 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:26:26.762 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:26.762 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:26.763 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:26.763 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:26.763 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:26.763 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:26.764 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:26.764 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:26.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:26.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:26.764 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:26:26.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:26.765 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:26:26.765 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:26:26.765 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:26:26.766 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:26.766 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:26:26.766 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.075268_690.1 2022-01-11T14:26:31.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:31.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:31.769 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:26:31.812 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:26:31.813 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:26:31.884 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:26:31.884 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:26:31.884 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:26:31.885 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:26:31.885 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:26:31.885 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:26:31.885 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:31.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:31.886 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:26:31.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:31.886 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:26:31.886 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:31.887 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:31.887 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.198209_694.1", 2022-01-11T14:26:31.887 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:31.887 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:26:31.888 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:31.888 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.198209Z", 2022-01-11T14:26:31.888 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:31.888 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:31.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:31.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:31.889 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:26:31.889 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:31.889 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:31.890 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:26:31.890 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:31.890 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:31.890 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:31.891 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:26:31.891 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:31.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:31.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:31.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:31.892 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:26:31.892 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:31.892 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:31.892 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:31.893 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:31.893 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:31.893 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:31.893 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:31.894 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:31.894 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:31.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:31.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:31.895 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:26:31.895 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:31.895 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:31.895 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:31.895 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:31.896 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:31.896 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:31.896 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:31.896 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:31.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:31.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:31.897 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:26:31.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:31.897 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:26:31.898 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:26:31.898 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:26:31.898 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:31.898 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:26:31.900 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.198209_694.1 2022-01-11T14:26:36.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:36.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:36.903 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:26:36.964 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:26:36.964 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:26:37.035 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:26:37.035 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:26:37.035 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:26:37.036 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:26:37.036 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:26:37.036 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:26:37.036 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:37.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:37.037 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:26:37.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:37.037 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:26:37.038 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:37.038 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:37.038 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.198209_694.1", 2022-01-11T14:26:37.038 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:37.039 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:26:37.039 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:37.040 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.198209Z", 2022-01-11T14:26:37.040 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:37.040 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:37.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:37.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:37.041 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:26:37.041 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:37.041 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:37.041 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:26:37.042 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:37.042 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:37.042 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:37.042 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:26:37.043 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:37.043 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:37.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:37.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:37.044 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:26:37.044 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:37.044 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:37.044 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:37.044 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:37.045 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:37.045 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:37.045 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:37.045 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:37.046 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:37.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:37.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:37.046 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:26:37.047 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:37.047 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:37.047 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:37.047 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:37.047 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:37.048 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:37.048 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:37.048 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:37.048 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:37.049 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:37.049 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:26:37.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:37.051 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:26:37.051 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:26:37.051 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:26:37.051 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:37.052 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:26:37.053 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.198209_694.1 2022-01-11T14:26:42.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:42.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:42.057 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:26:42.122 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:26:42.122 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:26:42.212 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:26:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:26:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:26:42.213 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:26:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:26:42.214 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:26:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:42.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:26:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:42.216 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:26:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:42.217 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.968115_681.1", 2022-01-11T14:26:42.218 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:26:42.219 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.968115Z", 2022-01-11T14:26:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:42.220 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:42.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:26:42.222 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:42.223 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:26:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:42.224 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:26:42.225 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:42.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:42.227 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:26:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:42.228 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:42.229 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:42.230 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:42.231 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:42.232 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:26:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:42.233 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:42.234 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:42.235 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:42.236 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:42.236 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:42.236 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:26:42.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:42.237 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:26:42.237 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:26:42.238 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:26:42.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:42.238 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:26:42.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.968115_681.1 2022-01-11T14:26:47.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:47.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:47.242 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:26:47.300 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:26:47.300 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:26:47.358 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:26:47.359 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:26:47.359 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:26:47.359 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:26:47.360 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:26:47.360 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:26:47.360 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:47.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:47.361 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:26:47.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:47.361 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:26:47.362 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:47.362 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:47.362 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.998061_684.1", 2022-01-11T14:26:47.362 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:47.363 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:26:47.363 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:47.363 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.998061Z", 2022-01-11T14:26:47.363 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:47.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:47.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:47.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:47.364 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:26:47.364 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:47.365 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:47.365 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:26:47.365 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:47.366 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:47.366 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:47.366 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:26:47.367 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:47.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:47.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:47.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:47.367 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:26:47.368 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:47.368 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:47.368 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:47.368 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:47.368 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:47.369 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:47.369 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:47.369 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:47.369 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:47.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:47.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:47.370 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:26:47.370 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:47.371 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:47.371 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:47.371 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:47.371 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:47.371 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:47.372 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:47.372 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:47.372 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:47.372 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:47.373 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:26:47.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:47.373 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:26:47.373 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:26:47.373 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:26:47.374 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:47.374 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:26:47.375 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.998061_684.1 2022-01-11T14:26:52.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:52.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:52.378 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:26:52.421 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:26:52.422 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:26:52.500 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:26:52.500 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:26:52.500 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:26:52.501 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:26:52.501 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:26:52.501 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:26:52.501 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:52.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:52.502 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:26:52.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:52.502 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:26:52.502 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:52.503 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:52.503 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.075268_690.1", 2022-01-11T14:26:52.503 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:52.503 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:26:52.504 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:52.504 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.075268Z", 2022-01-11T14:26:52.504 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:52.504 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:52.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:52.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:52.505 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:26:52.505 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:52.506 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:52.506 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:26:52.506 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:52.506 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:52.506 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:52.507 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:26:52.507 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:52.507 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:52.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:52.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:52.508 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:26:52.508 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:52.508 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:52.508 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:52.509 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:52.509 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:52.509 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:52.509 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:52.509 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:52.510 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:52.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:52.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:52.510 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:26:52.511 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:52.511 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:52.511 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:52.511 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:52.511 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:52.512 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:52.512 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:52.512 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:52.512 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:52.512 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:52.513 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:26:52.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:52.513 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:26:52.513 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:26:52.514 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:26:52.514 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:52.514 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:26:52.515 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.075268_690.1 2022-01-11T14:26:57.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:57.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:26:57.518 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:26:57.588 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:26:57.588 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:26:57.730 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:26:57.730 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:26:57.731 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:26:57.731 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:26:57.731 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:26:57.731 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:26:57.732 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:57.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:57.732 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:26:57.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:57.733 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:26:57.733 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:57.733 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:57.733 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.024470_689.1", 2022-01-11T14:26:57.733 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:57.734 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:26:57.734 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:57.734 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.024470Z", 2022-01-11T14:26:57.734 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:57.735 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:57.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:57.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:57.735 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:26:57.736 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:57.736 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:57.736 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:26:57.736 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:57.736 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:57.737 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:57.737 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:26:57.737 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:57.737 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:57.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:57.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:57.738 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:26:57.738 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:57.739 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:57.739 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:57.739 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:57.739 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:57.739 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:57.740 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:57.740 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:57.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:57.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:57.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:26:57.741 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:26:57.741 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:26:57.741 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:26:57.741 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:26:57.742 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:26:57.742 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:26:57.742 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:26:57.742 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:26:57.743 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:26:57.743 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:57.743 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:57.743 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:26:57.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:26:57.744 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:26:57.744 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:26:57.744 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:26:57.744 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:26:57.745 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:26:57.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.024470_689.1 2022-01-11T14:27:02.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:02.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:02.749 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:27:02.821 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:27:02.822 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:27:03.012 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:27:03.013 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:27:03.013 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:27:03.013 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:27:03.014 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:27:03.014 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:27:03.014 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:03.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:03.015 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:27:03.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:03.015 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:27:03.015 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:03.015 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:03.016 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.230925_700.1", 2022-01-11T14:27:03.016 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:03.016 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:27:03.016 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:03.017 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.230925Z", 2022-01-11T14:27:03.017 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:03.017 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:03.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:03.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:03.018 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:27:03.019 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:03.019 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:03.019 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:27:03.019 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:27:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:27:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:27:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:27:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:27:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:27:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:27:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:03.028 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:27:03.029 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.230925_700.1 2022-01-11T14:27:08.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:08.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:08.032 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:27:08.112 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:27:08.112 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:27:08.240 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:27:08.240 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:27:08.241 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:27:08.241 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:27:08.241 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:27:08.241 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:27:08.241 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:08.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:08.242 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:27:08.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:08.243 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:27:08.243 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:08.243 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:08.243 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.965074_680.1", 2022-01-11T14:27:08.244 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:08.244 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:27:08.244 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:08.244 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.965074Z", 2022-01-11T14:27:08.245 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:08.245 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:08.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:08.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:08.245 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:27:08.246 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:08.246 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:08.246 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:27:08.246 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:08.247 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:08.247 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:08.247 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:27:08.247 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:08.248 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:08.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:08.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:08.248 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:27:08.248 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:08.249 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:08.249 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:08.249 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:08.249 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:08.249 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:08.250 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:08.250 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:08.250 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:08.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:08.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:08.251 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:27:08.251 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:08.251 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:08.251 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:08.252 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:08.252 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:08.252 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:08.252 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:08.252 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:08.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:08.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:08.253 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:27:08.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:08.254 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:27:08.254 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:27:08.254 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:27:08.254 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:08.255 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:27:08.256 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.965074_680.1 2022-01-11T14:27:13.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:13.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:13.259 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:27:13.320 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:27:13.321 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:27:13.412 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:27:13.412 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:27:13.413 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:27:13.413 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:27:13.413 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:27:13.413 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:27:13.413 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:13.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:13.414 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:27:13.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:13.414 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:27:13.415 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:13.415 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:13.415 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.998061_684.1", 2022-01-11T14:27:13.415 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:13.416 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:27:13.416 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:13.416 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.998061Z", 2022-01-11T14:27:13.416 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:13.417 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:13.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:13.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:13.417 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:27:13.418 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:13.418 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:13.418 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:27:13.418 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:13.418 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:13.419 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:13.419 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:27:13.419 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:13.420 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:13.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:13.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:13.420 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:27:13.421 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:13.421 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:13.421 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:13.421 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:13.421 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:13.422 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:13.422 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:13.422 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:13.423 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:13.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:13.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:13.424 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:27:13.425 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:13.425 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:13.425 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:13.426 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:13.426 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:13.426 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:13.427 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:13.427 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:13.427 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:13.428 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:13.428 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:27:13.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:13.430 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:27:13.430 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:27:13.430 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:27:13.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:13.431 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:27:13.432 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.998061_684.1 2022-01-11T14:27:18.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:18.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:18.435 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:27:18.500 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:27:18.500 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:27:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:27:18.588 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:27:18.588 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:27:18.589 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:27:18.589 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:27:18.589 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:27:18.590 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:18.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:18.590 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:27:18.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:18.591 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:27:18.591 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:18.592 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:18.592 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.198209_694.1", 2022-01-11T14:27:18.592 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:18.592 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:27:18.592 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:18.593 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.198209Z", 2022-01-11T14:27:18.593 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:18.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:18.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:18.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:18.594 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:27:18.594 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:18.595 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:18.595 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:27:18.595 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:18.595 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:18.595 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:18.596 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:27:18.596 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:18.596 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:18.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:18.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:18.597 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:27:18.597 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:18.598 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:18.598 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:18.598 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:18.598 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:18.599 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:18.599 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:18.599 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:18.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:18.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:18.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:18.601 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:27:18.601 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:18.601 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:18.601 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:18.602 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:18.602 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:18.602 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:18.603 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:18.603 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:18.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:18.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:18.603 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:27:18.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:18.604 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:27:18.604 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:27:18.604 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:27:18.605 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:18.605 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:27:18.606 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.198209_694.1 2022-01-11T14:27:23.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:23.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:23.609 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:27:23.655 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:27:23.655 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:27:23.720 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:27:23.720 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:27:23.721 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:27:23.721 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:27:23.721 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:27:23.721 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:27:23.722 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:23.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:23.722 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:27:23.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:23.723 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:27:23.723 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:23.723 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:23.723 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:27:23.723 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:23.724 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:27:23.724 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:23.724 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:27:23.724 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:23.725 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:23.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:23.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:23.725 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:27:23.725 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:23.726 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:23.726 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:27:23.726 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:23.726 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:23.727 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:23.727 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:27:23.727 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:23.727 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:23.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:23.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:23.728 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:27:23.728 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:23.729 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:23.729 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:23.729 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:23.729 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:23.729 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:23.730 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:23.730 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:23.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:23.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:23.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:23.731 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:27:23.731 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:23.731 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:23.731 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:23.732 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:23.732 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:23.732 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:23.732 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:23.733 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:23.733 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:23.733 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:23.733 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:27:23.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:23.734 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:27:23.734 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:27:23.734 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:27:23.734 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:23.735 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:27:23.736 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.258675_705.1 2022-01-11T14:27:28.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:28.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:28.739 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:27:28.816 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:27:28.816 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:27:28.941 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:27:28.942 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:27:28.942 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:27:28.943 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:27:28.943 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:27:28.943 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:27:28.943 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:28.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:28.944 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:27:28.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:28.944 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:27:28.944 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:28.945 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:28.945 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:27:28.945 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:28.946 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:27:28.946 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:28.947 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:27:28.947 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:28.947 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:28.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:28.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:28.948 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:27:28.948 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:28.948 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:28.948 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:27:28.949 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:28.949 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:28.949 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:28.949 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:27:28.949 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:28.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:28.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:28.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:28.950 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:27:28.951 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:28.951 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:28.951 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:28.951 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:28.951 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:28.952 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:28.953 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:28.953 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:28.953 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:28.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:28.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:28.954 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:27:28.954 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:28.955 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:28.955 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:28.955 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:28.955 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:28.956 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:28.956 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:28.956 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:28.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:28.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:28.957 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:27:28.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:28.957 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:27:28.957 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:27:28.957 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:27:28.958 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:28.958 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:27:28.959 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.952442_679.1 2022-01-11T14:27:33.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:33.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:33.962 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:27:34.029 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:27:34.029 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:27:34.346 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:27:34.346 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:27:34.346 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:27:34.347 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:27:34.347 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:27:34.347 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:27:34.347 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:34.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:34.348 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:27:34.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:34.348 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:27:34.349 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:34.349 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:34.349 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:27:34.349 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:34.349 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:27:34.350 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:34.350 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:27:34.350 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:34.350 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:34.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:34.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:34.351 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:27:34.351 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:34.352 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:34.352 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:27:34.352 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:34.352 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:34.352 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:34.353 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:27:34.353 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:34.353 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:34.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:34.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:34.354 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:27:34.354 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:34.355 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:34.355 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:34.355 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:34.355 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:34.356 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:34.356 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:34.356 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:34.356 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:34.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:34.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:34.357 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:27:34.357 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:34.357 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:34.358 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:34.358 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:34.358 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:34.358 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:34.359 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:34.359 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:34.359 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:34.359 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:34.360 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:27:34.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:34.360 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:27:34.360 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:27:34.360 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:27:34.361 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:34.361 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:27:34.361 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.952442_679.1 2022-01-11T14:27:39.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:39.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:39.493 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:27:39.568 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:27:39.568 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:27:39.712 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:27:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:27:39.713 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:27:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:27:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:27:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:27:39.714 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:39.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:39.715 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:27:39.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:39.715 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:27:39.716 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:39.716 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:39.716 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.973179_683.1", 2022-01-11T14:27:39.716 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:39.716 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:27:39.717 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:39.717 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.973179Z", 2022-01-11T14:27:39.717 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:39.717 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:39.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:39.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:39.718 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:27:39.719 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:39.719 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:39.719 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:27:39.719 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:39.720 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:39.720 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:39.720 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:27:39.721 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:39.721 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:39.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:39.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:39.722 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:27:39.722 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:39.722 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:39.722 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:39.722 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:39.723 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:39.723 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:39.723 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:39.723 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:39.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:39.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:39.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:39.725 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:27:39.725 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:39.725 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:39.725 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:39.725 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:39.726 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:39.726 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:39.726 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:39.726 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:39.727 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:39.727 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:39.727 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:27:39.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:39.727 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:27:39.728 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:27:39.728 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:27:39.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:39.729 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:27:39.730 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.973179_683.1 2022-01-11T14:27:44.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:44.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:44.733 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:27:44.819 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:27:44.819 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:27:44.997 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:27:44.998 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:27:44.999 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:27:44.999 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:27:45.000 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:27:45.000 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:27:45.000 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:45.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:45.001 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:27:45.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:45.002 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:27:45.002 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:45.003 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:45.003 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.004662_687.1", 2022-01-11T14:27:45.004 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:45.005 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:27:45.005 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:45.005 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.004662Z", 2022-01-11T14:27:45.006 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:45.006 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:45.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:45.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:45.007 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:27:45.007 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:45.007 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:45.007 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:27:45.008 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:45.008 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:45.008 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:45.008 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:27:45.009 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:45.009 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:45.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:45.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:45.009 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:27:45.010 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:45.010 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:45.010 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:45.010 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:45.011 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:45.011 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:45.011 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:45.011 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:45.012 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:45.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:45.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:45.012 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:27:45.012 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:45.013 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:45.013 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:45.013 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:45.013 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:45.013 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:45.014 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:45.014 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:45.014 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:45.014 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:45.015 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:27:45.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:45.015 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:27:45.016 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:27:45.016 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:27:45.016 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:45.016 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:27:45.017 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.004662_687.1 2022-01-11T14:27:50.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:50.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:50.020 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:27:50.113 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:27:50.114 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:27:50.318 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:27:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:27:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:27:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:27:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:27:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:27:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:27:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:27:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.281170_707.1", 2022-01-11T14:27:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:27:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.281170Z", 2022-01-11T14:27:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:27:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:27:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:27:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:27:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:27:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:27:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:27:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:27:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:27:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:50.334 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:27:50.335 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.281170_707.1 2022-01-11T14:27:55.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:55.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:27:55.339 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:27:55.414 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:27:55.415 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:27:55.564 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:27:55.564 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:27:55.564 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:27:55.565 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:27:55.565 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:27:55.565 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:27:55.565 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:55.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:55.566 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:27:55.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:55.566 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:27:55.566 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:55.567 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:55.567 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.075268_690.1", 2022-01-11T14:27:55.567 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:55.567 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:27:55.567 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:55.568 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.075268Z", 2022-01-11T14:27:55.568 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:55.568 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:55.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:55.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:55.569 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:27:55.569 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:55.569 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:55.569 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:27:55.570 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:55.570 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:55.570 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:55.570 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:27:55.570 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:55.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:55.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:55.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:55.571 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:27:55.571 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:55.572 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:55.572 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:55.572 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:55.572 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:55.573 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:55.573 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:55.573 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:55.573 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:55.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:55.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:27:55.574 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:27:55.574 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:27:55.574 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:27:55.574 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:27:55.575 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:27:55.575 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:27:55.575 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:27:55.575 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:27:55.576 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:27:55.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:55.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:55.576 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:27:55.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:27:55.577 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:27:55.577 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:27:55.577 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:27:55.577 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:27:55.577 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:27:55.579 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.075268_690.1 2022-01-11T14:28:00.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:00.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:00.581 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:28:00.925 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:28:00.925 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:28:00.981 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:28:00.981 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:28:00.981 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:28:00.982 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:28:00.982 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:28:00.982 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:28:00.982 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:00.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:00.983 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:28:00.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:00.983 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:28:00.984 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:00.984 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:00.984 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.002077_686.1", 2022-01-11T14:28:00.984 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:00.985 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:28:00.985 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:00.985 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.002077Z", 2022-01-11T14:28:00.985 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:00.986 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:00.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:00.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:00.986 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:28:00.986 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:00.987 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:00.987 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:28:00.987 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:00.987 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:00.988 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:00.988 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:28:00.988 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:00.988 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:00.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:00.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:00.989 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:28:00.989 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:00.990 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:00.990 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:00.990 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:00.990 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:00.991 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:00.991 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:00.991 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:00.991 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:00.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:00.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:00.992 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:28:00.992 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:00.993 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:00.993 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:00.993 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:00.993 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:00.993 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:00.994 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:00.994 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:00.994 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:00.994 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:00.995 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:28:00.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:00.995 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:28:00.995 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:28:00.996 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:28:00.996 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:00.996 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:28:00.997 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.002077_686.1 2022-01-11T14:28:05.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:05.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:06.000 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:28:06.052 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:28:06.053 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:28:06.120 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:28:06.120 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:28:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:28:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:28:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:28:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:28:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:28:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:28:06.123 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:06.123 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:06.123 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.969581_682.1", 2022-01-11T14:28:06.123 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:06.124 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:28:06.124 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:06.124 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.969581Z", 2022-01-11T14:28:06.125 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:06.125 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:06.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:06.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:28:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:28:06.128 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:06.128 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:06.128 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:06.128 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:28:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:06.130 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:28:06.130 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:06.130 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:06.130 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:28:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:06.135 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:06.135 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:06.135 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:28:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:28:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:28:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:28:06.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:06.138 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:28:06.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.969581_682.1 2022-01-11T14:28:11.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:11.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:11.143 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:28:11.232 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:28:11.233 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:28:11.329 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:28:11.329 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:28:11.330 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:28:11.330 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:28:11.331 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:28:11.331 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:28:11.331 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:11.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:11.332 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:28:11.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:11.332 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:28:11.332 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:11.332 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:11.333 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.968115_681.1", 2022-01-11T14:28:11.333 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:11.333 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:28:11.334 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:11.334 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.968115Z", 2022-01-11T14:28:11.334 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:11.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:11.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:11.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:11.336 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:28:11.337 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:11.337 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:11.337 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:28:11.338 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:11.338 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:11.339 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:11.339 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:28:11.340 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:11.340 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:11.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:11.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:11.341 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:28:11.342 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:11.342 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:11.343 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:11.343 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:11.344 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:11.344 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:11.344 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:11.345 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:11.345 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:11.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:11.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:11.347 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:28:11.347 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:11.348 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:11.348 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:11.348 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:11.349 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:11.349 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:11.350 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:11.350 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:11.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:11.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:11.352 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:28:11.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:11.352 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:28:11.353 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:28:11.353 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:28:11.354 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:11.355 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:28:11.357 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.968115_681.1 2022-01-11T14:28:16.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:16.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:16.360 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:28:16.428 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:28:16.428 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:28:16.553 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:28:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:28:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:28:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:28:16.554 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:28:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:28:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:16.555 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:28:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:28:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:16.556 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.969581_682.1", 2022-01-11T14:28:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:28:16.557 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.969581Z", 2022-01-11T14:28:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:16.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:28:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:16.559 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:28:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:16.560 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:28:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:16.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:28:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:16.562 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:16.563 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:28:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:16.565 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:16.566 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:28:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:16.567 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:28:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:28:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:28:16.568 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:16.568 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:28:16.570 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.969581_682.1 2022-01-11T14:28:21.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:21.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:21.574 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:28:21.653 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:28:21.653 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:28:21.874 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:28:21.874 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:28:21.875 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:28:21.875 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:28:21.876 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:28:21.876 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:28:21.876 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:21.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:21.877 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:28:21.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:21.877 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:28:21.877 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:21.878 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:21.878 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.020042_688.1", 2022-01-11T14:28:21.878 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:21.878 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:28:21.878 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:21.879 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.020042Z", 2022-01-11T14:28:21.879 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:21.879 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:21.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:21.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:21.880 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:28:21.880 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:21.880 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:21.881 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:28:21.881 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:21.881 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:21.881 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:21.882 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:28:21.882 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:21.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:21.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:21.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:21.883 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:28:21.883 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:21.883 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:21.884 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:21.884 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:21.884 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:21.884 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:21.885 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:21.885 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:21.885 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:21.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:21.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:21.886 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:28:21.886 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:21.886 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:21.887 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:21.887 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:21.887 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:21.887 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:21.888 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:21.888 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:21.888 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:21.888 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:21.889 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:28:21.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:21.889 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:28:21.889 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:28:21.889 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:28:21.890 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:21.890 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:28:21.891 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.020042_688.1 2022-01-11T14:28:26.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:26.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:26.894 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:28:26.972 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:28:26.973 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:28:27.103 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:28:27.103 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:28:27.103 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:28:27.103 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:28:27.104 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:28:27.104 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:28:27.105 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:27.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:27.105 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:28:27.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:27.105 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:28:27.106 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:27.106 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:27.106 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.075268_690.1", 2022-01-11T14:28:27.106 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:27.107 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:28:27.107 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:27.107 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.075268Z", 2022-01-11T14:28:27.107 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:27.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:27.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:27.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:27.109 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:28:27.109 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:27.110 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:27.110 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:28:27.110 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:27.110 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:27.110 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:27.111 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:28:27.111 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:27.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:27.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:27.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:27.113 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:28:27.114 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:27.114 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:27.114 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:27.114 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:27.115 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:27.115 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:27.115 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:27.115 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:27.115 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:27.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:27.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:27.116 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:28:27.116 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:27.117 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:27.117 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:27.117 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:27.117 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:27.117 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:27.118 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:27.118 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:27.118 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:27.119 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:27.119 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:28:27.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:27.120 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:28:27.120 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:28:27.120 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:28:27.120 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:27.121 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:28:27.122 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.075268_690.1 2022-01-11T14:28:30.664 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:28:30.562+0000 7f3d897c4700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2022-01-11T14:28:32.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:32.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:32.126 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:28:32.198 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:28:32.199 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:28:32.377 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:28:32.378 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:28:32.378 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:28:32.378 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:28:32.379 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:28:32.379 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:28:32.379 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:32.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:32.379 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:28:32.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:32.380 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:28:32.380 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:32.380 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:32.381 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.138030_693.1", 2022-01-11T14:28:32.381 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:32.381 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:28:32.381 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:32.382 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.138030Z", 2022-01-11T14:28:32.382 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:32.382 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:32.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:32.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:32.383 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:28:32.383 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:32.383 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:32.383 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:28:32.384 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:32.385 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:32.385 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:32.385 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:28:32.386 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:32.386 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:32.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:32.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:32.387 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:28:32.387 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:32.387 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:32.387 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:32.387 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:32.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:32.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:32.389 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:28:32.389 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:32.390 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:32.390 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:32.390 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:32.390 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:32.391 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:32.391 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:32.391 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:32.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:32.392 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:32.392 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:28:32.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:32.392 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:28:32.392 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:28:32.393 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:28:32.393 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:32.393 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:28:32.395 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.138030_693.1 2022-01-11T14:28:37.396 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:37.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:37.398 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:28:37.476 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:28:37.477 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:28:37.650 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:28:37.985 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:28:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:28:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:28:37.986 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:28:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:28:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:37.987 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:28:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:28:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:37.988 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.075268_690.1", 2022-01-11T14:28:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:28:37.989 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.075268Z", 2022-01-11T14:28:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:37.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:28:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:37.991 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:28:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:37.992 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:28:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:37.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:28:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:37.994 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:37.995 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:37.996 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:28:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:37.998 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:37.999 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:28:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:28:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:28:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:28:38.000 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:38.001 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:28:38.002 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.075268_690.1 2022-01-11T14:28:43.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:43.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:43.005 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:28:43.059 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:28:43.059 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:28:43.122 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:28:43.122 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:28:43.123 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:28:43.123 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:28:43.123 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:28:43.123 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:28:43.124 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:43.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:43.124 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:28:43.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:43.125 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:28:43.125 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:43.125 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:43.125 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.209246_697.1", 2022-01-11T14:28:43.125 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:43.126 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:28:43.126 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:43.126 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.209246Z", 2022-01-11T14:28:43.127 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:43.127 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:43.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:43.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:43.128 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:28:43.128 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:43.128 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:43.129 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:28:43.129 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:43.129 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:43.129 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:43.130 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:28:43.130 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:43.130 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:43.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:43.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:43.131 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:28:43.131 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:43.131 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:43.131 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:43.132 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:43.132 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:43.132 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:43.133 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:43.133 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:43.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:43.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:43.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:43.134 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:28:43.134 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:43.134 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:43.134 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:43.135 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:43.135 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:43.135 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:43.135 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:43.136 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:43.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:43.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:43.136 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:28:43.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:43.137 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:28:43.137 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:28:43.137 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:28:43.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:43.138 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:28:43.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910389.209246_697.1 2022-01-11T14:28:48.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:48.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:28:48.142 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:28:48.186 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:28:48.186 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:28:48.243 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:28:48.244 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:28:48.245 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:28:48.245 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:28:48.245 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:28:48.245 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:28:48.245 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:48.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:48.246 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:28:48.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:48.246 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:28:48.247 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:48.247 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:48.247 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910388.952442_679.1", 2022-01-11T14:28:48.247 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:48.248 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:28:48.248 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:48.248 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:08.952442Z", 2022-01-11T14:28:48.248 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:48.249 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:48.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:48.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:48.249 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:28:48.249 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:48.250 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:48.250 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:28:48.250 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:48.250 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:48.251 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:48.251 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:28:48.251 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:48.251 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:48.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:48.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:48.252 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:28:48.252 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:48.252 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:48.252 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:48.253 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:48.253 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:48.253 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:48.253 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:48.254 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:48.254 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:48.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:48.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:48.254 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:28:48.255 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:48.255 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:28:48.255 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:28:48.255 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:28:48.256 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:28:48.256 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:28:48.256 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:28:48.256 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:48.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:48.257 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:48.257 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:28:48.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:48.257 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:28:48.257 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:28:48.258 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:28:48.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:48.258 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:28:48.259 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911018.227948_778.1 target=1_1641910388.952442_679.1 2022-01-11T14:28:50.610 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:28:50.611+0000 7f3d897c4700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2022-01-11T14:28:53.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2022-01-11T14:28:53.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:28:53.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:28:53.264 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2022-01-11T14:28:53.314 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:28:53.314 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:28:53.338 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:28:53.340+0000 7f9752b28500 0 failed reading obj info from .rgw.root:zone_info.776aa573-af61-46ca-a8eb-de18a933b620: (2) No such file or directory 2022-01-11T14:28:53.339 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:28:53.340+0000 7f9752b28500 0 WARNING: could not read zone params for zone id=776aa573-af61-46ca-a8eb-de18a933b620 name=test-zone1 2022-01-11T14:28:53.339 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:28:53.340+0000 7f9752b28500 0 failed reading obj info from .rgw.root:zone_info.a285481c-4c5c-487e-978c-9cbad192af6e: (2) No such file or directory 2022-01-11T14:28:53.339 INFO:teuthology.orchestra.run.smithi172.stderr:2022-01-11T14:28:53.340+0000 7f9752b28500 0 WARNING: could not read zone params for zone id=a285481c-4c5c-487e-978c-9cbad192af6e name=test-zone3 2022-01-11T14:28:53.343 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:28:53.343 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:28:53.343 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone2", 2022-01-11T14:28:53.343 INFO:teuthology.orchestra.run.smithi172.stdout: "domain_root": "test-zone2.rgw.meta:root", 2022-01-11T14:28:53.344 INFO:teuthology.orchestra.run.smithi172.stdout: "control_pool": "test-zone2.rgw.control", 2022-01-11T14:28:53.344 INFO:teuthology.orchestra.run.smithi172.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2022-01-11T14:28:53.344 INFO:teuthology.orchestra.run.smithi172.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2022-01-11T14:28:53.344 INFO:teuthology.orchestra.run.smithi172.stdout: "log_pool": "test-zone2.rgw.log", 2022-01-11T14:28:53.345 INFO:teuthology.orchestra.run.smithi172.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2022-01-11T14:28:53.345 INFO:teuthology.orchestra.run.smithi172.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2022-01-11T14:28:53.345 INFO:teuthology.orchestra.run.smithi172.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2022-01-11T14:28:53.345 INFO:teuthology.orchestra.run.smithi172.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2022-01-11T14:28:53.346 INFO:teuthology.orchestra.run.smithi172.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2022-01-11T14:28:53.346 INFO:teuthology.orchestra.run.smithi172.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2022-01-11T14:28:53.346 INFO:teuthology.orchestra.run.smithi172.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2022-01-11T14:28:53.346 INFO:teuthology.orchestra.run.smithi172.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2022-01-11T14:28:53.347 INFO:teuthology.orchestra.run.smithi172.stdout: "otp_pool": "test-zone2.rgw.otp", 2022-01-11T14:28:53.347 INFO:teuthology.orchestra.run.smithi172.stdout: "system_key": { 2022-01-11T14:28:53.347 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "QXCZ5A4RM0GLA2FE", 2022-01-11T14:28:53.348 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "Vw8mmsFBJWE37n1A3UI8CDKu5VBidxxi" 2022-01-11T14:28:53.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:53.348 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_pools": [ 2022-01-11T14:28:53.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:53.348 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "default-placement", 2022-01-11T14:28:53.349 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:28:53.349 INFO:teuthology.orchestra.run.smithi172.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2022-01-11T14:28:53.349 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": { 2022-01-11T14:28:53.349 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD": { 2022-01-11T14:28:53.350 INFO:teuthology.orchestra.run.smithi172.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2022-01-11T14:28:53.350 INFO:teuthology.orchestra.run.smithi172.stdout: "compression_type": "random" 2022-01-11T14:28:53.350 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:53.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:53.351 INFO:teuthology.orchestra.run.smithi172.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2022-01-11T14:28:53.351 INFO:teuthology.orchestra.run.smithi172.stdout: "index_type": 0 2022-01-11T14:28:53.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:53.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:53.351 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:28:53.352 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:28:53.352 INFO:teuthology.orchestra.run.smithi172.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2022-01-11T14:28:53.352 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:28:53.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:28:53.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:28:53.355 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2022-01-11T14:28:53.397 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:28:53.397 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:28:53.449 INFO:teuthology.orchestra.run.smithi172.stderr:Sending period to new master zone 776aa573-af61-46ca-a8eb-de18a933b620 2022-01-11T14:28:54.486 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:28:54.486 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:28:54.487 INFO:teuthology.orchestra.run.smithi172.stdout: "epoch": 5, 2022-01-11T14:28:54.487 INFO:teuthology.orchestra.run.smithi172.stdout: "predecessor_uuid": "bef33aea-c28e-4d3d-b156-c57e2ca93938", 2022-01-11T14:28:54.487 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": [], 2022-01-11T14:28:54.488 INFO:teuthology.orchestra.run.smithi172.stdout: "period_map": { 2022-01-11T14:28:54.488 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:28:54.488 INFO:teuthology.orchestra.run.smithi172.stdout: "zonegroups": [ 2022-01-11T14:28:54.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:54.489 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "test-zonegroup", 2022-01-11T14:28:54.489 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zonegroup", 2022-01-11T14:28:54.489 INFO:teuthology.orchestra.run.smithi172.stdout: "api_name": "", 2022-01-11T14:28:54.489 INFO:teuthology.orchestra.run.smithi172.stdout: "is_master": "true", 2022-01-11T14:28:54.490 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2022-01-11T14:28:54.490 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:28:54.490 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:28:54.490 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames": [], 2022-01-11T14:28:54.491 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames_s3website": [], 2022-01-11T14:28:54.491 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:28:54.491 INFO:teuthology.orchestra.run.smithi172.stdout: "zones": [ 2022-01-11T14:28:54.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:54.492 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:28:54.492 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone2", 2022-01-11T14:28:54.492 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2022-01-11T14:28:54.492 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2022-01-11T14:28:54.492 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:28:54.493 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": "false", 2022-01-11T14:28:54.493 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": "true", 2022-01-11T14:28:54.493 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:28:54.493 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": "false", 2022-01-11T14:28:54.494 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2022-01-11T14:28:54.494 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": "false", 2022-01-11T14:28:54.494 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2022-01-11T14:28:54.494 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "" 2022-01-11T14:28:54.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:54.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:54.495 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:28:54.496 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone1", 2022-01-11T14:28:54.496 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2022-01-11T14:28:54.496 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:28:54.496 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:28:54.497 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": "false", 2022-01-11T14:28:54.497 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": "true", 2022-01-11T14:28:54.497 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:28:54.497 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": "false", 2022-01-11T14:28:54.498 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2022-01-11T14:28:54.498 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": "true", 2022-01-11T14:28:54.498 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2022-01-11T14:28:54.498 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "" 2022-01-11T14:28:54.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:54.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:54.499 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "a285481c-4c5c-487e-978c-9cbad192af6e", 2022-01-11T14:28:54.499 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone3", 2022-01-11T14:28:54.500 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2022-01-11T14:28:54.500 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi161.front.sepia.ceph.com:8001" 2022-01-11T14:28:54.500 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:28:54.500 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": "false", 2022-01-11T14:28:54.500 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": "true", 2022-01-11T14:28:54.501 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:28:54.501 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": "false", 2022-01-11T14:28:54.501 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2022-01-11T14:28:54.501 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": "true", 2022-01-11T14:28:54.502 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2022-01-11T14:28:54.502 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "" 2022-01-11T14:28:54.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:54.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:54.503 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "ccb5abc6-5490-49c7-b386-02859e29748c", 2022-01-11T14:28:54.503 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-zone4", 2022-01-11T14:28:54.503 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2022-01-11T14:28:54.504 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2022-01-11T14:28:54.504 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:28:54.504 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": "false", 2022-01-11T14:28:54.505 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": "true", 2022-01-11T14:28:54.505 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:28:54.505 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": "false", 2022-01-11T14:28:54.505 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "pubsub", 2022-01-11T14:28:54.506 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": "false", 2022-01-11T14:28:54.506 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [ 2022-01-11T14:28:54.506 INFO:teuthology.orchestra.run.smithi172.stdout: "test-zone1" 2022-01-11T14:28:54.507 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:28:54.507 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "" 2022-01-11T14:28:54.507 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:54.507 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:28:54.507 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_targets": [ 2022-01-11T14:28:54.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:54.508 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default-placement", 2022-01-11T14:28:54.508 INFO:teuthology.orchestra.run.smithi172.stdout: "tags": [], 2022-01-11T14:28:54.508 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": [ 2022-01-11T14:28:54.509 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD" 2022-01-11T14:28:54.509 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:28:54.509 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:54.509 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:28:54.509 INFO:teuthology.orchestra.run.smithi172.stdout: "default_placement": "default-placement", 2022-01-11T14:28:54.510 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:28:54.510 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_policy": { 2022-01-11T14:28:54.510 INFO:teuthology.orchestra.run.smithi172.stdout: "groups": [] 2022-01-11T14:28:54.510 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:54.511 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:54.511 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2022-01-11T14:28:54.511 INFO:teuthology.orchestra.run.smithi172.stdout: "short_zone_ids": [ 2022-01-11T14:28:54.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:54.513 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:28:54.513 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 2774529435 2022-01-11T14:28:54.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:54.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:54.514 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:28:54.514 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 618671346 2022-01-11T14:28:54.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:54.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:54.515 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "a285481c-4c5c-487e-978c-9cbad192af6e", 2022-01-11T14:28:54.515 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 544948410 2022-01-11T14:28:54.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:54.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:28:54.516 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "ccb5abc6-5490-49c7-b386-02859e29748c", 2022-01-11T14:28:54.516 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 2309622666 2022-01-11T14:28:54.517 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:54.517 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:28:54.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:54.517 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zonegroup": "test-zonegroup", 2022-01-11T14:28:54.518 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:28:54.518 INFO:teuthology.orchestra.run.smithi172.stdout: "period_config": { 2022-01-11T14:28:54.518 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_quota": { 2022-01-11T14:28:54.518 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2022-01-11T14:28:54.519 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2022-01-11T14:28:54.519 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2022-01-11T14:28:54.519 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2022-01-11T14:28:54.519 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2022-01-11T14:28:54.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:54.520 INFO:teuthology.orchestra.run.smithi172.stdout: "user_quota": { 2022-01-11T14:28:54.520 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2022-01-11T14:28:54.520 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2022-01-11T14:28:54.520 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2022-01-11T14:28:54.521 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2022-01-11T14:28:54.521 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2022-01-11T14:28:54.521 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:28:54.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:28:54.522 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:28:54.522 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_name": "test-realm", 2022-01-11T14:28:54.523 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:28:54.523 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:28:54.523 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2022-01-11T14:29:54.527 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-45 2022-01-11T14:29:54.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:29:54.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:29:54.746 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:29:54.829 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:29:54.830 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:29:54.922 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:29:54.959 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:29:54.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:29:54.960 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641911394.664935_798.1", 2022-01-11T14:29:54.961 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:29:54.664935Z" 2022-01-11T14:29:54.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:29:54.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:29:54.962 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:29:54.962 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:29:54.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:29:54.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:29:54.963 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:29:54.963 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:29:54.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:29:54.963 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:29:54.963 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:29:54.964 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:29:54.964 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:29:54.964 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:29:54.965 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:29:54.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:29:54.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:29:54.967 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:29:55.019 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:29:55.019 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:29:55.099 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:29:55.100 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:29:55.100 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:29:55.100 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:29:55.100 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:29:55.100 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:29:55.101 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:29:55.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:29:55.101 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:29:55.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:29:55.102 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:29:55.102 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:29:55.102 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:29:55.102 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.254940_703.1", 2022-01-11T14:29:55.103 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:29:55.103 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:29:55.103 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:29:55.103 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.254940Z", 2022-01-11T14:29:55.103 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:29:55.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:29:55.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:29:55.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:29:55.104 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:29:55.105 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:29:55.105 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:29:55.105 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:29:55.105 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:29:55.105 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:29:55.106 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:29:55.106 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:29:55.106 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:29:55.106 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:29:55.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:29:55.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:29:55.107 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:29:55.107 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:29:55.107 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:29:55.108 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:29:55.108 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:29:55.108 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:29:55.108 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:29:55.109 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:29:55.110 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:29:55.110 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:29:55.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:29:55.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:29:55.111 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:29:55.111 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:29:55.111 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:29:55.112 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:29:55.112 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:29:55.112 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:29:55.112 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:29:55.113 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:29:55.113 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:29:55.113 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:29:55.113 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:29:55.113 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:29:55.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:29:55.114 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:29:55.114 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:29:55.114 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:29:55.115 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:29:55.115 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:29:55.116 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.254940_703.1 2022-01-11T14:30:00.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:00.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:00.120 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:30:00.166 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:30:00.167 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:30:00.237 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:30:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:30:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:30:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:30:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:30:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:30:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:30:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:30:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.254940_703.1", 2022-01-11T14:30:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:30:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.254940Z", 2022-01-11T14:30:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:30:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:30:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:30:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:30:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:30:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:30:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:30:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:30:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:30:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:00.252 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:30:00.253 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.254940_703.1 2022-01-11T14:30:05.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:05.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:05.280 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:30:05.337 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:30:05.337 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:30:05.402 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:30:05.402 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:30:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:30:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:30:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:30:05.403 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:30:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:30:05.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:30:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:05.405 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.254940_703.1", 2022-01-11T14:30:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:30:05.406 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.254940Z", 2022-01-11T14:30:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:05.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:30:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:05.408 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:30:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:05.409 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:30:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:05.410 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:30:05.411 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:05.413 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:30:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:05.415 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:05.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:30:05.417 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:05.418 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:30:05.420 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.254940_703.1 2022-01-11T14:30:10.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:10.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:10.423 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:30:10.474 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:30:10.475 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:30:10.546 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:30:10.546 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:30:10.547 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:30:10.547 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:30:10.547 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:30:10.547 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:30:10.547 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:10.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:10.548 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:30:10.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:10.548 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:30:10.549 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:10.549 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:10.549 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.254940_703.1", 2022-01-11T14:30:10.549 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:10.549 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:30:10.550 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:10.550 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.254940Z", 2022-01-11T14:30:10.550 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:10.550 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:10.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:10.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:10.551 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:30:10.551 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:10.551 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:10.552 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:30:10.552 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:10.552 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:10.552 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:10.553 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:30:10.553 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:10.553 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:10.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:10.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:10.554 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:30:10.554 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:10.554 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:10.554 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:10.555 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:10.555 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:10.555 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:10.555 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:10.555 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:10.556 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:10.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:10.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:10.556 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:30:10.556 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:10.557 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:10.557 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:10.557 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:10.557 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:10.558 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:10.558 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:10.558 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:10.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:10.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:10.559 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:30:10.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:10.559 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:30:10.559 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:30:10.560 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:30:10.560 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:10.560 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:30:10.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.254940_703.1 2022-01-11T14:30:15.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:15.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:15.564 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:30:15.613 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:30:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:30:15.677 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:30:15.678 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:30:15.678 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:30:15.678 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:30:15.679 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:30:15.679 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:30:15.679 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:15.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:15.680 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:30:15.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:15.680 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:30:15.680 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:15.681 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:15.681 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.254940_703.1", 2022-01-11T14:30:15.681 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:15.681 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:30:15.682 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:15.682 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.254940Z", 2022-01-11T14:30:15.682 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:15.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:15.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:15.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:15.683 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:30:15.683 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:15.684 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:15.684 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:30:15.684 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:15.684 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:15.685 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:15.685 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:30:15.685 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:15.686 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:15.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:15.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:15.686 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:30:15.687 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:15.687 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:15.687 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:15.687 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:15.688 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:15.688 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:15.688 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:15.688 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:15.688 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:15.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:15.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:15.689 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:30:15.689 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:15.690 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:15.690 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:15.690 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:15.690 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:15.691 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:15.691 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:15.691 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:15.692 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:15.692 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:15.692 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:30:15.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:15.692 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:30:15.693 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:30:15.693 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:30:15.693 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:15.693 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:30:15.695 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.254940_703.1 2022-01-11T14:30:20.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:20.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:20.698 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:30:20.747 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:30:20.747 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:30:20.825 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:30:20.826 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:20.828 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.254940_703.1", 2022-01-11T14:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:20.829 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.254940Z", 2022-01-11T14:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:20.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:20.831 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:20.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:20.837 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:20.838 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:30:20.839 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:20.840 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:20.841 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:20.841 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:20.841 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:20.841 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:20.841 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:20.842 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:30:20.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:20.842 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:30:20.842 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:30:20.842 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:30:20.843 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:20.843 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:30:20.845 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.254940_703.1 2022-01-11T14:30:25.773 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:30:25.774+0000 7f3da3411700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2022-01-11T14:30:25.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:25.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:25.847 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:30:25.897 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:30:25.897 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:30:25.985 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:30:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:30:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:30:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:30:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:30:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:30:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:25.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:25.989 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:30:25.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:30:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.254940_703.1", 2022-01-11T14:30:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:30:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.254940Z", 2022-01-11T14:30:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:25.993 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:25.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:25.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:25.993 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:30:25.994 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:25.994 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:25.994 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:30:25.994 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:25.995 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:25.995 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:25.995 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:30:25.995 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:25.995 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:25.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:25.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:25.996 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:30:25.997 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:25.997 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:25.997 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:25.998 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:25.998 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:25.998 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:25.998 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:25.999 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:25.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:26.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:26.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:26.000 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:30:26.001 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:26.001 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:26.001 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:26.001 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:26.001 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:26.002 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:26.002 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:26.002 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:26.002 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:26.003 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:26.003 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:30:26.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:26.003 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:30:26.004 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:30:26.004 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:30:26.004 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:26.004 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:30:26.006 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.254940_703.1 2022-01-11T14:30:31.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:31.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:31.009 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:30:31.060 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:30:31.060 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:30:31.136 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:30:31.137 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:30:31.137 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:30:31.137 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:30:31.138 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:30:31.138 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:30:31.138 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:31.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:31.139 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:30:31.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:31.139 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:30:31.139 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:31.140 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:31.140 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.254940_703.1", 2022-01-11T14:30:31.140 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:31.140 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:30:31.141 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:31.141 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.254940Z", 2022-01-11T14:30:31.141 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:31.141 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:31.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:31.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:31.142 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:30:31.142 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:31.143 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:31.143 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:30:31.143 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:31.143 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:31.144 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:31.144 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:30:31.144 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:31.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:31.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:31.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:31.145 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:30:31.146 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:31.146 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:31.146 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:31.146 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:31.147 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:31.147 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:31.147 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:31.148 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:31.148 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:31.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:31.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:31.149 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:30:31.149 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:31.149 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:31.149 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:31.149 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:31.150 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:31.150 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:31.150 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:31.150 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:31.151 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:31.151 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:31.151 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:30:31.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:31.152 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:30:31.152 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:30:31.152 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:30:31.152 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:31.153 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:30:31.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.254940_703.1 2022-01-11T14:30:36.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:36.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:36.157 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:30:36.211 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:30:36.211 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:30:36.274 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:30:36.274 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:30:36.274 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:30:36.275 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:30:36.275 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:30:36.276 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:30:36.276 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:36.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:36.277 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:30:36.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:36.278 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:30:36.278 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:36.278 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:36.279 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.254940_703.1", 2022-01-11T14:30:36.279 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:36.279 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:30:36.279 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:36.280 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.254940Z", 2022-01-11T14:30:36.280 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:36.280 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:36.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:36.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:36.281 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:30:36.281 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:36.281 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:36.281 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:30:36.282 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:36.282 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:36.283 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:36.283 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:30:36.284 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:36.284 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:36.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:36.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:36.285 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:30:36.285 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:36.285 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:36.286 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:36.286 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:36.286 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:36.286 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:36.287 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:36.287 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:36.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:36.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:36.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:36.288 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:30:36.288 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:36.288 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:36.288 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:36.289 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:36.289 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:36.289 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:36.289 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:36.289 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:36.290 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:36.290 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:36.290 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:30:36.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:36.291 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:30:36.291 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:30:36.291 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:30:36.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:36.292 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:30:36.294 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.254940_703.1 2022-01-11T14:30:41.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:41.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:41.299 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:30:41.348 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:30:41.349 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:30:41.419 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:30:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:30:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:30:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:30:41.420 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:30:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:30:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:41.421 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:30:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:30:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.254940_703.1", 2022-01-11T14:30:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:30:41.423 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.254940Z", 2022-01-11T14:30:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:41.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:30:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:41.425 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:30:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:41.426 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:30:41.427 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:41.427 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:41.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:41.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:41.427 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:30:41.428 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:41.428 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:41.428 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:41.428 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:41.429 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:41.429 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:41.429 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:41.429 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:41.429 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:41.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:41.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:41.430 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:30:41.430 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:41.431 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:41.431 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:41.431 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:41.431 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:41.432 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:41.432 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:41.432 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:41.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:41.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:41.433 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:30:41.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:41.433 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:30:41.433 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:30:41.434 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:30:41.434 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:41.434 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:30:41.435 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.254940_703.1 2022-01-11T14:30:46.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:46.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:46.438 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:30:46.488 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:30:46.488 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:30:46.549 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:30:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:30:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:30:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:30:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:30:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:30:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:30:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:30:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.254940_703.1", 2022-01-11T14:30:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:30:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.254940Z", 2022-01-11T14:30:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:46.555 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:30:46.555 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:46.555 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:46.555 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:30:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:30:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:30:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:46.559 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:46.559 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:46.559 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:46.559 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:46.559 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:30:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:30:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:30:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:30:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:30:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:46.564 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:30:46.565 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.254940_703.1 2022-01-11T14:30:51.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:51.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:51.568 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:30:51.617 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:30:51.618 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:30:51.694 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:30:51.694 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:30:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:30:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:30:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:30:51.695 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:30:51.696 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:51.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:51.696 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:30:51.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:51.697 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:30:51.697 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:51.697 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:51.697 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.254940_703.1", 2022-01-11T14:30:51.698 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:51.698 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:30:51.698 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:51.698 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.254940Z", 2022-01-11T14:30:51.699 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:51.699 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:51.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:51.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:51.700 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:30:51.700 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:51.700 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:51.701 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:30:51.701 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:51.701 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:51.701 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:51.701 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:30:51.702 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:51.702 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:51.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:51.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:51.703 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:30:51.703 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:51.703 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:51.703 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:51.704 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:51.704 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:51.704 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:51.704 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:51.705 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:51.705 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:51.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:51.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:51.706 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:30:51.706 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:51.707 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:51.707 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:51.707 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:51.707 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:51.707 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:51.708 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:51.708 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:51.708 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:51.708 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:51.709 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:30:51.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:51.709 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:30:51.709 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:30:51.709 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:30:51.710 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:51.710 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:30:51.711 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.254940_703.1 2022-01-11T14:30:56.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:56.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:30:56.715 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:30:56.759 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:30:56.759 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:30:56.826 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:30:56.826 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:30:56.826 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:30:56.826 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:30:56.827 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:30:56.827 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:30:56.827 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:56.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:56.828 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:30:56.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:56.828 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:30:56.828 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:56.829 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:56.829 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.285437_708.1", 2022-01-11T14:30:56.829 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:56.829 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:30:56.829 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:56.830 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.285437Z", 2022-01-11T14:30:56.830 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:56.830 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:56.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:56.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:56.831 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:30:56.831 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:56.832 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:56.832 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:30:56.832 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:56.832 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:56.832 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:56.833 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:30:56.833 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:56.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:56.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:56.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:56.834 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:30:56.834 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:56.834 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:56.834 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:56.835 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:56.835 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:56.835 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:56.835 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:56.836 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:56.836 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:56.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:56.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:30:56.837 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:30:56.837 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:30:56.837 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:30:56.838 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:30:56.838 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:30:56.838 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:30:56.839 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:30:56.839 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:30:56.839 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:30:56.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:56.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:56.840 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:30:56.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:30:56.840 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:30:56.841 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:30:56.841 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:30:56.841 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:30:56.841 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:30:56.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.285437_708.1 2022-01-11T14:31:01.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:01.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:01.845 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:31:01.893 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:31:01.893 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:31:02.097 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:31:02.098 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:02.099 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:31:02.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:02.101 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.261451_706.1", 2022-01-11T14:31:02.102 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:02.103 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.261451Z", 2022-01-11T14:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:02.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:02.105 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:31:02.106 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:02.107 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:02.108 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:02.109 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:31:02.113 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:02.114 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:02.115 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:02.116 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:02.117 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:31:02.119 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.261451_706.1 2022-01-11T14:31:07.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:07.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:07.122 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:31:07.184 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:31:07.185 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:31:07.255 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:31:07.256 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:31:07.257 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:31:07.257 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:31:07.257 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:31:07.258 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:31:07.258 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:07.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:07.259 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:31:07.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:07.260 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:31:07.260 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:07.261 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:07.262 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.281170_707.1", 2022-01-11T14:31:07.262 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:07.263 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:31:07.264 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:07.264 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.281170Z", 2022-01-11T14:31:07.264 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:07.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:07.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:07.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:07.266 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:31:07.267 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:07.267 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:07.268 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:31:07.268 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:07.268 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:07.269 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:07.269 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:31:07.271 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:07.271 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:07.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:07.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:07.274 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:31:07.274 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:07.275 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:07.275 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:07.275 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:07.275 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:07.276 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:07.276 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:07.276 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:07.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:07.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:07.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:07.277 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:31:07.277 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:07.277 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:07.278 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:07.278 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:07.278 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:07.278 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:07.279 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:07.279 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:07.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:07.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:07.280 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:31:07.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:07.280 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:31:07.280 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:31:07.280 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:31:07.281 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:07.281 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:31:07.282 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.281170_707.1 2022-01-11T14:31:12.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:12.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:12.285 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:31:12.328 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:31:12.328 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:31:12.408 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:31:12.409 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:31:12.409 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:31:12.410 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:31:12.410 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:31:12.410 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:31:12.410 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:12.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:12.411 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:31:12.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:12.411 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:31:12.412 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:12.412 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:12.412 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:31:12.412 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:12.412 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:31:12.413 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:12.413 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:31:12.413 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:12.413 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:12.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:12.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:12.414 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:31:12.414 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:12.415 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:12.415 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:31:12.415 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:12.415 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:12.416 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:12.416 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:31:12.416 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:12.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:12.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:12.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:12.417 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:31:12.417 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:12.417 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:12.418 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:12.418 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:12.418 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:12.418 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:12.419 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:12.419 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:12.420 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:12.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:12.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:12.420 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:31:12.421 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:12.421 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:12.421 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:12.421 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:12.422 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:12.422 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:12.422 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:12.423 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:12.423 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:12.423 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:12.423 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:31:12.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:12.424 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:31:12.424 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:31:12.424 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:31:12.424 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:12.425 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:31:12.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:31:17.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:17.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:17.429 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:31:17.479 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:31:17.479 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:31:17.540 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:31:17.541 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:31:17.541 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:31:17.541 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:31:17.541 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:31:17.542 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:31:17.542 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:17.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:17.542 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:31:17.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:17.543 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:31:17.543 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:17.543 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:17.544 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.519237_721.1", 2022-01-11T14:31:17.544 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:17.544 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:31:17.544 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:17.545 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.519237Z", 2022-01-11T14:31:17.545 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:17.545 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:17.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:17.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:17.546 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:31:17.546 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:17.546 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:17.547 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:31:17.547 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:17.547 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:17.547 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:17.548 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:31:17.548 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:17.548 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:17.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:17.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:17.549 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:31:17.549 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:17.549 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:17.552 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:31:17.552 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:17.552 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:17.553 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:17.553 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:17.553 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:17.554 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:17.554 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:17.554 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:17.554 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:17.555 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:17.555 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:31:17.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:17.555 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:31:17.555 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:31:17.556 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:31:17.556 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:17.556 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:31:17.557 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.519237_721.1 2022-01-11T14:31:22.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:22.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:22.560 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:31:22.607 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:31:22.607 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:31:22.666 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:31:22.667 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:31:22.667 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:31:22.667 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:31:22.667 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:31:22.668 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:31:22.668 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:22.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:22.668 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:31:22.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:22.669 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:31:22.669 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:22.670 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:22.670 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.281170_707.1", 2022-01-11T14:31:22.670 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:22.670 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:31:22.671 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:22.671 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.281170Z", 2022-01-11T14:31:22.671 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:22.672 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:22.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:22.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:22.672 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:31:22.673 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:22.673 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:22.673 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:31:22.673 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:22.674 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:22.674 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:22.674 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:31:22.674 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:22.675 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:22.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:22.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:22.675 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:31:22.676 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:22.676 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:22.676 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:22.676 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:22.676 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:22.677 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:22.677 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:22.677 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:22.677 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:22.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:22.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:22.678 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:31:22.679 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:22.679 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:22.679 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:22.679 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:22.680 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:22.680 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:22.680 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:22.681 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:22.681 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:22.681 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:22.681 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:31:22.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:22.682 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:31:22.682 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:31:22.682 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:31:22.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:22.683 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:31:22.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.281170_707.1 2022-01-11T14:31:27.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:27.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:27.687 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:31:27.765 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:31:27.766 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:31:28.114 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:31:28.114 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:31:28.114 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:31:28.115 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:31:28.115 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:31:28.115 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:31:28.115 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:28.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:28.116 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:31:28.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:28.116 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:31:28.116 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:28.117 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:28.117 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:31:28.117 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:28.117 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:31:28.117 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:28.118 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:31:28.118 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:28.118 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:28.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:28.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:28.119 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:31:28.119 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:28.119 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:28.119 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:31:28.120 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:28.120 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:28.120 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:28.120 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:31:28.121 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:28.121 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:28.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:28.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:28.121 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:31:28.122 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:28.122 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:28.123 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:28.123 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:28.123 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:28.123 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:28.123 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:28.124 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:28.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:28.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:28.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:28.124 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:31:28.125 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:28.125 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:28.125 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:28.125 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:28.126 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:28.126 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:28.126 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:28.126 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:28.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:28.127 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:28.127 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:31:28.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:28.127 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:31:28.128 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:31:28.128 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:31:28.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:28.128 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:31:28.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:31:33.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:33.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:33.132 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:31:33.189 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:31:33.189 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:31:33.326 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:31:33.326 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:31:33.327 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:31:33.327 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:31:33.327 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:31:33.327 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:31:33.327 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:33.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:33.328 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:31:33.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:33.328 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:31:33.329 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:33.329 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:33.329 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:31:33.329 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:33.329 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:31:33.330 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:33.330 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:31:33.330 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:33.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:33.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:33.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:33.331 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:31:33.331 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:33.332 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:33.332 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:31:33.332 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:33.332 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:33.332 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:33.333 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:31:33.333 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:33.333 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:33.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:33.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:33.334 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:31:33.334 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:33.334 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:33.335 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:33.335 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:33.335 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:33.335 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:33.336 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:33.336 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:33.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:33.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:33.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:33.337 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:31:33.337 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:33.337 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:33.337 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:33.338 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:33.338 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:33.338 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:33.338 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:33.338 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:33.339 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:33.339 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:33.339 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:31:33.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:33.340 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:31:33.340 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:31:33.340 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:31:33.340 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:33.340 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:31:33.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:31:38.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:38.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:38.344 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:31:38.390 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:31:38.391 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:31:38.446 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:31:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:31:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:31:38.447 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:31:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:31:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:31:38.448 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:31:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:38.449 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:31:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.261451_706.1", 2022-01-11T14:31:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:38.450 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:31:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.261451Z", 2022-01-11T14:31:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:38.451 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:31:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:38.452 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:31:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:38.453 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:31:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:38.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:31:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:38.455 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:38.456 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:38.457 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:31:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:38.458 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:38.459 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:38.460 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:38.460 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:31:38.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:38.460 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:31:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:31:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:31:38.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:38.461 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:31:38.463 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.261451_706.1 2022-01-11T14:31:43.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:43.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:43.465 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:31:43.516 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:31:43.517 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:31:43.580 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:31:43.581 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:31:43.581 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:31:43.581 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:31:43.582 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:31:43.582 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:31:43.582 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:43.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:43.583 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:31:43.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:43.584 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:31:43.584 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:43.585 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:43.585 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:31:43.585 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:43.586 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:31:43.586 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:43.587 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:31:43.587 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:43.587 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:43.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:43.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:43.588 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:31:43.589 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:43.589 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:43.589 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:31:43.589 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:43.590 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:43.590 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:43.590 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:31:43.590 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:43.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:43.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:43.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:43.591 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:31:43.591 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:43.592 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:43.592 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:43.592 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:43.592 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:43.593 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:43.593 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:43.593 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:43.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:43.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:43.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:43.594 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:31:43.595 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:43.595 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:43.595 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:43.596 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:43.596 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:43.596 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:43.597 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:43.597 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:43.598 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:43.598 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:43.598 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:31:43.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:43.599 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:31:43.599 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:31:43.600 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:31:43.600 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:43.600 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:31:43.602 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:31:48.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:48.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:48.605 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:31:48.654 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:31:48.655 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:31:48.725 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:31:48.725 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:31:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:31:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:31:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:31:48.726 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:31:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:31:48.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:31:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.404090_712.1", 2022-01-11T14:31:48.728 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:31:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.404090Z", 2022-01-11T14:31:48.729 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:48.730 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:31:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:31:48.731 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:31:48.732 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:31:48.733 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:48.734 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:48.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:31:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:48.736 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:48.737 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:48.738 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:31:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:31:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:31:48.739 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:31:48.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:48.740 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:31:48.741 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.404090_712.1 2022-01-11T14:31:53.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:53.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:53.746 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:31:53.839 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:31:53.840 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:31:53.963 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:31:53.964 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:31:53.964 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:31:53.964 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:31:53.964 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:31:53.965 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:31:53.965 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:53.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:53.965 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:31:53.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:53.966 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:31:53.966 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:53.966 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:53.967 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:31:53.967 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:53.967 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:31:53.968 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:53.968 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:31:53.968 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:53.968 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:53.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:53.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:53.969 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:31:53.969 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:53.969 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:53.970 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:31:53.970 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:53.970 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:53.970 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:53.971 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:31:53.971 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:53.971 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:53.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:53.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:53.972 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:31:53.972 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:53.972 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:53.972 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:53.973 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:53.973 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:53.973 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:53.973 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:53.974 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:53.974 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:53.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:53.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:53.975 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:31:53.975 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:53.975 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:53.975 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:53.975 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:53.976 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:53.976 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:53.976 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:53.976 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:53.976 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:53.977 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:53.977 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:31:53.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:53.977 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:31:53.978 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:31:53.978 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:31:53.978 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:53.978 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:31:53.980 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:31:58.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:58.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:31:58.983 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:31:59.034 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:31:59.035 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:31:59.279 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:31:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:31:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:31:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:31:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:31:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:31:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:31:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:31:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.344534_710.1", 2022-01-11T14:31:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:31:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.344534Z", 2022-01-11T14:31:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:31:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:31:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:31:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:31:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:59.290 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:59.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:59.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:31:59.290 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:31:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:31:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:31:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:31:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:31:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:31:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:31:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:31:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:31:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:31:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:31:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:31:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:31:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:31:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:31:59.294 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:31:59.295 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.344534_710.1 2022-01-11T14:32:04.297 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:04.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:04.298 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:32:04.342 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:32:04.343 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:32:04.417 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:32:04.417 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:32:04.417 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:32:04.418 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:32:04.418 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:32:04.418 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:32:04.418 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:04.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:04.419 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:32:04.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:04.419 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:32:04.420 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:04.420 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:04.420 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.519237_721.1", 2022-01-11T14:32:04.420 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:04.421 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:32:04.421 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:04.421 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.519237Z", 2022-01-11T14:32:04.421 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:04.422 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:04.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:04.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:04.422 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:32:04.423 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:04.423 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:04.423 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:32:04.423 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:04.424 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:04.424 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:04.424 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:32:04.424 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:04.425 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:04.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:04.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:04.425 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:32:04.426 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:04.426 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:04.426 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:04.427 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:04.427 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:04.427 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:04.427 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:04.427 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:04.428 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:04.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:04.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:04.428 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:32:04.429 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:04.429 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:04.429 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:04.429 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:04.430 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:04.430 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:04.430 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:04.430 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:04.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:04.431 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:04.431 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:32:04.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:04.431 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:32:04.432 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:32:04.432 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:32:04.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:04.432 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:32:04.433 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.519237_721.1 2022-01-11T14:32:09.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:09.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:09.436 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:32:09.487 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:32:09.488 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:32:09.550 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:32:09.550 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:32:09.550 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:32:09.551 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:32:09.551 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:32:09.551 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:32:09.551 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:09.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:09.552 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:32:09.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:09.552 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:32:09.553 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:09.553 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:09.553 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:32:09.554 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:09.554 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:32:09.554 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:09.555 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:32:09.555 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:09.555 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:09.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:09.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:09.556 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:32:09.556 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:09.556 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:09.556 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:32:09.557 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:09.557 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:09.557 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:09.557 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:32:09.558 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:09.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:09.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:09.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:09.559 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:32:09.559 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:09.559 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:09.560 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:09.560 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:09.560 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:09.560 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:09.561 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:09.561 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:09.561 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:09.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:09.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:09.562 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:32:09.562 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:09.562 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:09.562 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:09.563 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:09.563 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:09.563 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:09.563 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:09.564 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:09.564 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:09.564 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:09.565 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:32:09.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:09.565 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:32:09.565 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:32:09.565 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:32:09.566 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:09.566 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:32:09.567 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:32:14.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:14.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:14.571 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:32:14.619 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:32:14.619 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:32:14.682 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:32:14.682 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:32:14.682 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:32:14.683 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:32:14.683 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:32:14.683 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:32:14.683 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:14.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:14.684 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:32:14.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:14.684 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:32:14.685 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:14.685 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:14.685 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:32:14.685 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:14.686 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:32:14.686 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:14.686 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:32:14.686 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:14.686 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:14.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:14.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:14.687 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:32:14.687 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:14.688 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:14.688 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:32:14.688 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:14.688 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:14.688 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:14.689 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:32:14.689 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:14.689 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:14.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:14.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:14.690 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:32:14.690 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:14.690 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:14.690 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:14.691 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:14.691 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:14.691 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:14.691 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:14.692 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:14.692 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:14.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:14.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:14.693 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:32:14.693 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:14.693 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:14.693 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:14.693 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:14.694 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:14.694 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:14.694 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:14.694 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:14.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:14.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:14.695 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:32:14.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:14.696 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:32:14.696 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:32:14.696 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:32:14.697 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:14.697 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:32:14.699 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:32:19.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:19.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:19.702 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:32:19.772 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:32:19.772 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:32:20.188 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:32:20.189 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:32:20.189 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:32:20.189 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:32:20.190 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:32:20.190 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:32:20.190 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:20.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:20.191 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:32:20.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:20.192 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:32:20.192 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:20.192 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:20.192 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:32:20.193 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:20.193 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:32:20.193 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:20.193 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:32:20.194 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:20.194 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:20.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:20.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:20.196 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:32:20.196 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:20.197 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:20.197 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:32:20.197 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:20.198 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:20.199 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:20.199 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:32:20.199 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:20.200 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:20.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:20.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:20.201 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:32:20.201 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:20.202 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:20.202 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:20.202 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:20.203 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:20.203 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:20.203 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:20.204 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:20.204 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:20.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:20.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:20.207 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:32:20.207 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:20.207 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:20.207 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:20.207 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:20.208 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:20.208 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:20.208 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:20.208 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:20.209 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:20.209 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:20.209 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:32:20.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:20.210 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:32:20.210 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:32:20.210 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:32:20.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:20.210 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:32:20.211 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:32:25.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:25.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:25.214 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:32:25.267 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:32:25.268 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:32:25.410 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:32:25.411 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:32:25.411 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:32:25.412 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:32:25.412 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:32:25.412 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:32:25.413 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:25.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:25.413 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:32:25.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:25.414 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:32:25.414 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:25.415 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:25.415 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.261451_706.1", 2022-01-11T14:32:25.415 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:25.415 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:32:25.415 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:25.416 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.261451Z", 2022-01-11T14:32:25.416 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:25.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:25.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:25.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:25.417 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:32:25.417 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:25.419 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:25.419 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:32:25.419 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:25.419 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:25.420 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:25.420 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:32:25.420 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:25.420 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:25.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:25.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:25.421 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:32:25.421 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:25.422 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:25.422 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:25.422 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:25.422 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:25.423 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:25.423 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:25.423 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:25.423 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:25.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:25.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:25.424 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:32:25.424 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:25.425 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:25.425 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:25.425 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:25.425 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:25.426 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:25.426 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:25.426 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:25.427 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:25.427 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:25.427 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:32:25.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:25.428 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:32:25.428 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:32:25.428 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:32:25.429 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:25.429 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:32:25.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.261451_706.1 2022-01-11T14:32:30.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:30.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:30.433 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:32:30.478 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:32:30.478 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:32:30.553 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:32:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:32:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:32:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:32:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:32:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:32:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:32:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:32:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.485421_716.1", 2022-01-11T14:32:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:32:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.485421Z", 2022-01-11T14:32:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:32:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:32:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:32:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:32:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:32:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:32:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:32:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:32:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:32:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:30.568 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:32:30.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.485421_716.1 2022-01-11T14:32:35.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:35.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:35.571 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:32:35.616 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:32:35.616 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:32:35.697 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:32:35.697 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:32:35.697 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:32:35.698 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:32:35.698 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:32:35.698 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:32:35.698 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:35.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:35.699 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:32:35.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:35.699 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:32:35.699 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:35.700 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:35.700 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:32:35.700 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:35.700 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:32:35.701 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:35.701 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:32:35.701 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:35.701 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:35.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:35.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:35.702 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:32:35.702 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:35.702 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:35.703 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:32:35.703 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:35.703 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:35.703 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:35.703 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:32:35.704 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:35.704 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:35.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:35.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:35.705 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:32:35.705 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:35.705 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:35.705 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:35.705 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:35.706 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:35.706 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:35.706 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:35.706 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:35.707 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:35.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:35.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:35.707 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:32:35.707 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:35.708 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:35.708 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:35.708 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:35.708 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:35.709 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:35.709 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:35.709 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:35.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:35.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:35.710 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:32:35.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:35.710 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:32:35.710 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:32:35.711 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:32:35.711 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:35.711 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:32:35.712 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:32:40.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:40.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:40.715 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:32:40.761 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:32:40.761 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:32:40.828 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:32:40.829 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:32:40.829 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:32:40.829 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:32:40.830 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:32:40.830 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:32:40.830 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:40.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:40.830 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:32:40.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:40.831 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:32:40.832 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:40.832 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:40.832 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:32:40.832 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:40.833 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:32:40.833 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:40.833 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:32:40.833 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:40.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:40.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:40.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:40.834 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:32:40.834 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:40.835 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:40.835 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:32:40.835 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:40.835 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:40.836 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:40.836 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:32:40.836 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:40.836 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:40.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:40.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:40.837 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:32:40.837 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:40.837 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:40.838 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:40.838 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:40.838 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:40.839 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:40.839 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:40.840 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:40.841 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:40.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:40.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:40.841 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:32:40.842 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:40.842 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:40.842 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:40.842 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:40.843 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:40.843 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:40.843 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:40.843 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:40.844 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:40.844 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:40.844 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:32:40.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:40.844 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:32:40.845 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:32:40.845 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:32:40.845 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:40.845 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:32:40.847 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:32:45.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:45.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:45.850 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:32:45.902 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:32:45.903 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:32:46.101 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:32:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:32:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:32:46.102 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:32:46.103 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:32:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:32:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:46.104 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:32:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:32:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:46.105 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.485421_716.1", 2022-01-11T14:32:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:32:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:46.106 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.485421Z", 2022-01-11T14:32:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:46.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:32:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:32:46.108 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:32:46.109 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:46.110 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:32:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:46.111 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:46.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:46.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:46.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:46.113 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:32:46.113 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:46.113 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:46.114 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:46.114 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:46.114 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:46.114 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:46.115 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:46.115 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:46.115 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:46.115 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:46.116 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:32:46.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:46.116 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:32:46.116 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:32:46.116 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:32:46.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:46.117 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:32:46.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.485421_716.1 2022-01-11T14:32:51.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:51.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:51.121 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:32:51.172 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:32:51.172 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:32:51.230 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:32:51.231 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:32:51.231 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:32:51.232 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:32:51.232 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:32:51.232 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:32:51.232 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:51.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:51.233 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:32:51.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:51.233 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:32:51.234 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:51.234 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:51.234 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.510995_719.1", 2022-01-11T14:32:51.234 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:51.234 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:32:51.235 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:51.235 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.510995Z", 2022-01-11T14:32:51.235 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:51.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:51.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:51.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:51.236 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:32:51.236 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:51.237 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:51.237 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:32:51.238 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:51.238 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:51.238 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:51.239 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:32:51.240 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:51.240 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:51.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:51.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:51.241 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:32:51.242 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:51.242 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:51.243 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:51.243 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:51.243 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:51.243 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:51.243 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:51.244 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:51.244 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:51.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:51.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:51.245 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:32:51.245 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:51.245 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:51.245 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:51.245 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:51.246 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:51.246 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:51.246 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:51.246 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:51.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:51.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:51.247 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:32:51.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:51.247 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:32:51.248 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:32:51.248 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:32:51.248 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:51.248 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:32:51.250 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.510995_719.1 2022-01-11T14:32:56.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:56.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:32:56.252 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:32:56.324 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:32:56.324 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:32:56.515 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:32:56.515 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:32:56.515 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:32:56.515 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:32:56.516 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:32:56.516 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:32:56.516 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:56.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:56.517 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:32:56.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:56.517 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:32:56.517 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:56.517 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:56.518 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.519237_721.1", 2022-01-11T14:32:56.518 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:56.518 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:32:56.518 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:56.519 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.519237Z", 2022-01-11T14:32:56.519 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:56.519 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:56.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:56.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:56.520 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:32:56.520 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:56.520 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:56.520 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:32:56.521 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:56.521 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:56.521 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:56.521 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:32:56.522 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:56.522 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:56.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:56.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:56.522 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:32:56.523 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:56.523 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:56.523 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:56.523 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:56.524 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:56.524 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:56.524 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:56.524 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:56.524 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:56.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:56.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:32:56.525 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:32:56.525 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:32:56.526 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:32:56.526 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:32:56.526 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:32:56.526 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:32:56.526 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:32:56.527 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:32:56.527 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:32:56.527 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:56.528 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:56.528 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:32:56.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:32:56.528 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:32:56.528 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:32:56.529 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:32:56.529 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:32:56.529 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:32:56.530 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.519237_721.1 2022-01-11T14:33:01.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:01.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:01.533 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:33:01.577 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:33:01.577 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:33:01.657 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:33:01.657 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:33:01.657 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:33:01.658 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:33:01.658 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:33:01.658 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:33:01.658 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:01.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:01.659 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:33:01.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:01.659 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:33:01.660 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:01.660 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:01.660 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.344534_710.1", 2022-01-11T14:33:01.660 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:01.661 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:33:01.661 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:01.661 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.344534Z", 2022-01-11T14:33:01.661 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:01.661 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:01.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:01.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:01.662 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:33:01.662 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:01.663 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:01.663 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:33:01.663 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:01.663 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:01.664 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:01.664 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:33:01.664 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:01.664 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:01.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:01.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:01.665 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:33:01.665 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:01.666 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:01.666 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:01.666 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:01.666 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:01.667 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:01.667 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:01.667 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:01.667 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:01.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:01.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:01.668 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:33:01.668 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:01.668 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:01.669 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:01.669 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:01.669 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:01.669 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:01.670 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:01.670 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:01.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:01.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:01.670 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:33:01.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:01.671 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:33:01.671 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:33:01.671 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:33:01.672 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:01.672 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:33:01.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.344534_710.1 2022-01-11T14:33:06.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:06.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:06.676 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:33:06.726 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:33:06.727 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:33:06.807 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:33:06.807 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:33:06.807 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:33:06.808 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:33:06.808 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:33:06.808 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:33:06.808 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:06.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:06.809 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:33:06.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:06.809 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:33:06.810 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:06.810 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:06.810 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.257680_704.1", 2022-01-11T14:33:06.810 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:06.811 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:33:06.811 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:06.811 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.257680Z", 2022-01-11T14:33:06.811 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:06.811 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:06.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:06.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:06.812 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:33:06.812 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:06.813 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:06.813 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:33:06.813 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:06.814 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:06.814 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:06.814 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:33:06.815 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:06.815 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:06.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:06.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:06.816 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:33:06.816 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:06.816 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:06.816 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:06.817 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:06.817 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:06.817 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:06.817 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:06.818 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:06.818 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:06.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:06.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:06.819 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:33:06.819 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:06.819 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:06.819 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:06.819 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:06.820 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:06.820 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:06.820 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:06.820 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:06.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:06.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:06.821 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:33:06.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:06.822 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:33:06.822 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:33:06.822 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:33:06.822 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:06.823 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:33:06.824 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.257680_704.1 2022-01-11T14:33:11.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:11.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:11.827 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:33:11.878 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:33:11.878 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:33:11.952 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:33:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:33:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:33:11.952 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:33:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:33:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:33:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:11.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:33:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:33:11.954 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.485421_716.1", 2022-01-11T14:33:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:11.955 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:33:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.485421Z", 2022-01-11T14:33:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:11.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:33:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:11.957 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:33:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:11.958 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:33:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:11.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:33:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:11.960 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:11.961 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:11.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:33:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:11.963 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:11.964 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:33:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:11.965 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:33:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:33:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:33:11.966 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:11.966 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:33:11.968 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.485421_716.1 2022-01-11T14:33:16.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:16.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:16.970 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:33:17.055 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:33:17.055 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:33:17.283 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:33:17.284 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:33:17.284 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:33:17.284 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:33:17.285 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:33:17.285 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:33:17.285 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:17.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:17.286 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:33:17.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:17.286 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:33:17.286 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:17.287 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:17.287 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.476921_714.1", 2022-01-11T14:33:17.287 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:17.287 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:33:17.287 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:17.288 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.476921Z", 2022-01-11T14:33:17.288 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:17.288 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:17.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:17.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:17.289 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:33:17.289 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:17.289 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:17.289 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:33:17.290 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:17.290 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:17.290 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:17.290 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:33:17.291 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:17.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:17.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:17.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:17.292 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:33:17.292 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:17.292 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:17.293 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:17.293 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:17.293 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:17.293 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:17.294 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:17.294 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:17.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:17.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:17.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:17.295 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:33:17.295 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:17.295 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:17.295 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:17.296 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:17.296 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:17.296 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:17.297 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:17.298 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:17.298 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:17.298 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:17.298 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:33:17.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:17.299 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:33:17.299 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:33:17.299 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:33:17.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:17.300 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:33:17.301 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.476921_714.1 2022-01-11T14:33:22.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:22.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:22.304 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:33:22.356 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:33:22.357 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:33:22.578 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:33:22.578 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:33:22.578 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:33:22.579 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:33:22.579 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:33:22.579 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:33:22.579 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:22.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:22.580 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:33:22.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:22.580 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:33:22.580 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:22.581 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:22.581 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.261451_706.1", 2022-01-11T14:33:22.581 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:22.581 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:33:22.581 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:22.582 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.261451Z", 2022-01-11T14:33:22.582 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:22.582 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:22.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:22.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:22.583 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:33:22.583 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:22.583 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:22.583 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:33:22.584 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:22.584 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:22.584 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:22.584 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:33:22.585 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:22.585 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:22.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:22.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:22.586 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:33:22.586 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:22.586 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:22.586 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:22.587 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:22.587 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:22.587 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:22.587 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:22.587 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:22.588 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:22.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:22.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:22.588 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:33:22.589 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:22.589 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:22.589 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:22.589 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:22.589 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:22.590 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:22.590 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:22.590 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:22.590 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:22.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:22.591 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:33:22.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:22.591 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:33:22.591 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:33:22.592 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:33:22.592 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:22.592 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:33:22.593 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.261451_706.1 2022-01-11T14:33:27.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:27.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:27.596 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:33:27.641 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:33:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:33:27.697 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:33:27.698 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:33:27.698 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:33:27.698 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:33:27.699 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:33:27.699 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:33:27.699 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:27.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:27.700 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:33:27.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:27.700 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:33:27.701 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:27.701 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:27.701 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910391.794430_733.1", 2022-01-11T14:33:27.701 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:27.702 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:33:27.702 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:27.702 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:11.794430Z", 2022-01-11T14:33:27.702 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:27.702 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:27.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:27.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:27.703 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:33:27.703 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:27.704 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:27.704 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:33:27.704 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:27.704 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:27.705 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:27.705 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:33:27.705 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:27.705 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:27.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:27.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:27.706 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:33:27.706 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:27.707 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:27.707 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:27.707 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:27.707 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:27.708 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:27.708 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:27.708 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:27.708 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:27.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:27.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:27.709 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:33:27.709 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:27.709 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:33:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:33:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:33:27.713 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:33:27.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:27.713 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:33:27.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910391.794430_733.1 2022-01-11T14:33:32.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:32.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:32.717 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:33:32.765 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:33:32.766 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:33:32.834 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:33:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:33:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:33:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:33:32.835 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:33:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:33:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:32.836 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:33:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:33:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:32.837 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.257680_704.1", 2022-01-11T14:33:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:33:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:32.838 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.257680Z", 2022-01-11T14:33:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:32.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:33:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:33:32.840 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:33:32.841 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:32.842 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:33:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:32.843 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:32.844 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:33:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:32.845 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:32.846 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:32.847 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:33:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:33:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:33:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:33:32.849 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:32.849 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:33:32.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.257680_704.1 2022-01-11T14:33:37.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:37.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:37.854 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:33:37.937 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:33:37.938 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:33:38.338 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:33:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:33:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:33:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:33:38.339 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:33:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:33:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:33:38.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:33:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:38.341 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:33:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:33:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:38.342 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:33:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:38.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:33:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:38.344 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:33:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:33:38.345 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:33:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:38.347 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:33:38.349 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:38.350 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:38.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:33:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:33:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:33:38.352 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:33:38.353 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:38.353 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:33:38.354 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:33:43.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:43.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:43.356 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:33:43.399 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:33:43.399 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:33:43.508 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:33:43.508 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:33:43.509 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:33:43.509 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:33:43.509 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:33:43.509 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:33:43.510 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:43.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:43.510 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:33:43.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:43.511 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:33:43.511 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:43.511 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:43.511 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.257680_704.1", 2022-01-11T14:33:43.511 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:43.512 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:33:43.512 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:43.512 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.257680Z", 2022-01-11T14:33:43.512 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:43.513 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:43.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:43.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:43.513 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:33:43.513 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:43.514 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:43.514 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:33:43.514 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:43.514 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:43.515 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:43.515 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:33:43.515 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:43.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:43.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:43.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:43.516 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:33:43.516 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:43.516 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:43.517 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:43.517 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:43.517 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:43.517 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:43.517 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:43.518 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:43.518 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:43.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:43.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:43.519 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:33:43.519 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:43.519 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:43.519 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:43.520 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:43.520 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:43.520 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:43.520 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:43.521 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:43.521 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:43.521 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:43.521 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:33:43.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:43.522 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:33:43.522 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:33:43.522 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:33:43.522 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:43.523 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:33:43.524 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.257680_704.1 2022-01-11T14:33:48.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:48.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:48.527 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:33:48.591 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:33:48.592 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:33:48.672 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:33:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:33:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:33:48.672 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:33:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:33:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:33:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:48.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:33:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:33:48.674 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.408662_713.1", 2022-01-11T14:33:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:48.675 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:33:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.408662Z", 2022-01-11T14:33:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:48.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:33:48.677 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:33:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:48.678 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:33:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:48.679 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:33:48.680 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:48.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:33:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:48.683 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:48.684 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:33:48.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:33:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:33:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:33:48.686 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:48.686 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:33:48.687 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.408662_713.1 2022-01-11T14:33:53.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:53.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:53.691 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:33:53.735 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:33:53.736 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:33:53.806 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:33:53.806 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:33:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:33:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:33:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:33:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:33:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:33:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:33:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:33:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:33:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:33:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:33:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:33:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:33:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:33:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:33:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:33:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:33:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:33:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:33:53.822 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:53.822 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:33:53.823 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:33:58.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:58.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:33:58.826 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:33:58.874 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:33:58.874 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:33:58.941 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:33:58.942 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:33:58.942 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:33:58.942 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:33:58.943 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:33:58.943 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:33:58.943 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:58.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:58.943 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:33:58.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:58.944 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:33:58.944 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:58.945 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:58.945 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:33:58.945 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:58.945 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:33:58.946 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:58.946 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:33:58.946 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:58.946 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:58.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:58.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:58.947 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:33:58.947 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:58.947 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:58.948 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:33:58.948 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:58.948 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:58.948 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:58.949 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:33:58.949 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:58.949 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:58.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:58.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:58.950 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:33:58.950 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:58.950 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:58.950 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:58.951 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:58.951 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:58.951 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:58.951 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:58.951 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:58.952 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:58.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:58.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:33:58.952 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:33:58.953 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:33:58.953 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:33:58.953 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:33:58.953 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:33:58.954 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:33:58.954 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:33:58.954 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:33:58.954 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:33:58.955 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:58.955 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:58.955 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:33:58.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:33:58.956 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:33:58.956 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:33:58.956 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:33:58.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:33:58.957 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:33:58.958 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:34:03.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:03.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:03.961 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:34:04.011 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:34:04.012 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:34:04.130 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:34:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:34:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:34:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:34:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:34:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:34:04.132 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:04.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:04.132 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:34:04.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:04.133 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:34:04.133 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:04.133 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:04.133 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.285437_708.1", 2022-01-11T14:34:04.134 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:04.134 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:34:04.134 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:04.134 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.285437Z", 2022-01-11T14:34:04.135 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:04.135 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:04.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:04.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:04.136 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:34:04.136 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:04.136 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:34:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:34:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:34:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:34:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:04.143 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:04.143 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:04.143 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:04.143 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:04.143 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:34:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:34:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:34:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:34:04.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:04.145 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:34:04.146 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.285437_708.1 2022-01-11T14:34:09.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:09.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:09.149 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:34:09.233 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:34:09.234 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:34:09.388 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:34:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:34:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:34:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:34:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:34:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:34:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:34:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:34:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:34:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:34:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:34:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:34:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:34:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:34:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:34:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:34:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:34:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:34:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:34:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:34:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:09.403 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:34:09.405 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:34:14.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:14.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:14.407 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:34:14.478 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:34:14.478 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:34:14.604 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:34:14.605 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:34:14.605 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:34:14.605 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:34:14.605 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:34:14.606 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:34:14.606 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:14.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:14.607 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:34:14.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:14.607 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:34:14.608 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:14.608 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:14.608 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.404090_712.1", 2022-01-11T14:34:14.608 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:14.609 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:34:14.609 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:14.609 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.404090Z", 2022-01-11T14:34:14.609 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:14.610 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:14.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:14.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:14.610 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:34:14.611 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:14.611 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:14.611 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:34:14.611 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:14.612 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:14.612 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:14.612 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:34:14.612 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:14.612 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:14.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:14.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:14.613 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:34:14.613 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:14.614 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:14.614 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:14.614 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:14.614 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:14.615 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:14.615 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:14.615 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:14.615 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:14.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:14.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:14.616 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:34:14.616 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:14.617 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:14.617 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:14.617 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:14.617 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:14.617 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:14.618 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:14.618 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:14.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:14.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:14.619 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:34:14.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:14.619 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:34:14.619 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:34:14.619 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:34:14.620 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:14.620 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:34:14.620 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.404090_712.1 2022-01-11T14:34:19.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:19.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:19.623 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:34:19.667 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:34:19.667 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:34:19.746 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:34:19.746 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:34:19.746 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:34:19.747 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:34:19.747 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:34:19.747 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:34:19.747 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:19.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:19.748 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:34:19.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:19.748 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:34:19.749 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:19.749 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:19.749 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.257680_704.1", 2022-01-11T14:34:19.749 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:19.749 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:34:19.750 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:19.750 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.257680Z", 2022-01-11T14:34:19.750 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:19.750 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:19.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:19.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:19.751 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:34:19.751 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:19.752 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:19.752 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:34:19.752 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:19.752 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:19.753 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:19.753 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:34:19.753 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:19.753 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:19.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:19.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:19.754 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:34:19.754 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:19.754 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:19.755 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:19.755 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:19.755 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:19.755 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:19.756 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:19.756 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:19.756 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:19.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:19.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:19.757 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:34:19.757 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:19.757 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:19.757 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:19.758 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:19.758 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:19.758 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:19.758 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:19.758 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:19.759 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:19.759 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:19.759 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:34:19.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:19.760 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:34:19.760 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:34:19.760 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:34:19.760 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:19.760 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:34:19.762 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.257680_704.1 2022-01-11T14:34:24.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:24.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:24.765 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:34:24.810 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:34:24.811 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:34:24.867 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:34:24.867 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:34:24.867 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:34:24.868 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:34:24.868 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:34:24.868 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:34:24.868 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:24.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:24.869 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:34:24.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:24.869 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:34:24.869 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:24.870 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:24.870 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.485421_716.1", 2022-01-11T14:34:24.870 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:24.870 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:34:24.870 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:24.871 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.485421Z", 2022-01-11T14:34:24.871 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:24.871 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:24.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:24.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:24.872 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:34:24.872 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:24.872 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:24.872 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:34:24.873 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:24.873 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:24.873 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:24.873 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:34:24.873 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:24.874 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:24.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:24.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:24.874 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:34:24.875 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:24.875 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:24.875 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:24.875 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:24.875 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:24.876 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:24.876 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:24.876 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:24.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:24.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:24.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:24.877 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:34:24.878 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:24.878 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:24.878 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:24.878 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:24.879 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:24.879 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:24.879 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:24.879 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:24.879 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:24.880 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:24.880 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:34:24.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:24.880 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:34:24.881 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:34:24.881 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:34:24.881 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:24.881 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:34:24.882 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.485421_716.1 2022-01-11T14:34:29.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:29.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:29.885 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:34:29.933 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:34:29.933 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:34:30.004 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:34:30.004 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:34:30.004 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:34:30.005 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:34:30.005 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:34:30.005 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:34:30.005 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:30.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:30.006 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:34:30.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:30.007 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:34:30.007 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:30.007 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:30.008 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:34:30.008 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:30.008 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:34:30.009 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:30.009 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:34:30.010 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:30.010 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:30.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:30.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:30.011 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:34:30.011 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:30.011 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:30.011 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:34:30.012 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:30.012 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:30.012 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:30.013 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:34:30.013 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:30.013 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:30.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:30.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:30.014 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:34:30.014 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:30.015 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:30.015 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:30.015 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:30.016 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:30.016 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:30.016 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:30.016 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:30.017 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:30.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:30.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:30.017 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:34:30.017 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:30.018 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:30.019 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:30.019 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:30.020 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:30.020 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:30.020 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:30.020 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:30.020 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:30.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:30.021 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:34:30.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:30.021 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:34:30.022 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:34:30.022 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:34:30.022 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:30.023 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:34:30.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:34:35.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:35.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:35.027 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:34:35.097 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:34:35.097 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:34:35.488 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:34:35.488 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:34:35.489 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:34:35.489 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:34:35.489 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:34:35.489 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:34:35.490 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:35.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:35.490 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:34:35.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:35.491 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:34:35.491 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:35.491 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:35.491 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.340992_709.1", 2022-01-11T14:34:35.492 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:35.492 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:34:35.492 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:35.492 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.340992Z", 2022-01-11T14:34:35.492 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:35.493 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:35.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:35.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:35.493 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:34:35.494 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:35.494 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:35.494 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:34:35.494 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:35.495 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:35.495 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:35.495 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:34:35.495 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:35.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:35.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:35.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:35.496 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:34:35.497 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:35.497 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:35.497 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:35.497 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:35.497 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:35.498 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:35.498 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:35.498 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:35.498 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:35.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:35.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:35.499 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:34:35.499 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:35.500 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:35.500 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:35.500 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:35.500 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:35.500 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:35.501 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:35.501 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:35.501 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:35.501 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:35.502 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:34:35.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:35.502 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:34:35.502 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:34:35.502 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:34:35.503 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:35.503 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:34:35.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.340992_709.1 2022-01-11T14:34:40.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:40.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:40.507 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:34:40.577 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:34:40.577 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:34:40.778 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:34:40.778 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:34:40.779 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:34:40.779 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:34:40.779 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:34:40.779 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:34:40.779 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:40.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:40.780 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:34:40.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:40.780 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:34:40.781 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:40.781 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:40.781 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.285437_708.1", 2022-01-11T14:34:40.781 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:40.781 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:34:40.782 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:40.782 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.285437Z", 2022-01-11T14:34:40.782 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:40.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:40.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:40.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:40.783 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:34:40.783 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:40.784 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:40.784 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:34:40.784 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:40.784 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:40.784 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:40.785 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:34:40.785 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:40.785 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:40.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:40.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:40.786 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:34:40.786 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:40.786 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:40.786 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:40.787 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:40.787 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:40.787 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:40.787 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:40.787 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:40.788 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:40.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:40.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:40.788 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:34:40.789 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:40.789 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:40.789 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:40.789 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:40.789 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:40.790 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:40.790 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:40.790 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:40.790 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:40.791 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:40.791 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:34:40.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:40.791 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:34:40.791 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:34:40.792 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:34:40.792 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:40.792 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:34:40.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.285437_708.1 2022-01-11T14:34:45.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:45.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:45.796 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:34:45.841 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:34:45.841 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:34:45.897 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:34:45.897 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:34:45.897 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:34:45.898 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:34:45.898 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:34:45.898 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:34:45.898 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:45.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:45.903 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:34:45.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:45.903 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:34:45.904 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:45.904 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:45.904 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:34:45.904 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:45.905 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:34:45.905 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:45.905 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:34:45.905 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:45.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:45.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:45.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:45.906 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:34:45.907 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:45.907 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:45.907 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:34:45.907 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:45.908 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:45.908 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:45.908 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:34:45.908 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:45.909 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:45.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:45.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:45.909 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:34:45.909 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:45.910 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:45.910 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:45.910 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:45.910 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:45.911 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:45.911 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:45.911 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:45.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:45.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:45.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:45.912 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:34:45.912 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:45.912 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:45.913 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:45.913 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:45.913 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:45.913 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:45.914 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:45.914 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:45.914 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:45.914 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:45.914 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:34:45.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:45.915 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:34:45.915 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:34:45.915 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:34:45.916 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:45.916 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:34:45.917 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:34:50.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:50.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:50.921 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:34:50.970 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:34:50.971 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:34:51.056 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:34:51.057 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:34:51.057 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:34:51.057 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:34:51.057 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:34:51.058 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:34:51.058 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:51.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:51.058 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:34:51.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:51.059 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:34:51.059 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:51.059 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:51.060 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.519237_721.1", 2022-01-11T14:34:51.060 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:51.060 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:34:51.060 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:51.061 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.519237Z", 2022-01-11T14:34:51.061 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:51.061 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:51.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:51.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:51.062 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:34:51.062 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:51.062 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:51.062 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:34:51.063 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:51.063 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:51.063 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:51.063 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:34:51.063 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:51.064 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:51.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:51.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:51.064 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:34:51.064 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:51.065 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:51.065 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:51.065 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:51.065 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:51.066 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:51.066 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:51.066 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:51.066 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:51.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:51.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:51.067 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:34:51.067 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:51.067 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:51.067 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:51.068 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:51.068 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:51.068 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:51.068 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:51.069 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:51.069 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:51.069 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:51.069 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:34:51.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:51.070 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:34:51.070 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:34:51.070 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:34:51.071 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:51.071 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:34:51.072 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.519237_721.1 2022-01-11T14:34:56.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:56.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:34:56.074 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:34:56.119 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:34:56.120 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:34:56.205 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:34:56.206 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:34:56.206 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:34:56.206 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:34:56.206 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:34:56.207 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:34:56.207 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:56.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:56.207 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:34:56.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:56.208 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:34:56.208 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:56.208 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:56.209 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.491733_717.1", 2022-01-11T14:34:56.209 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:56.209 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:34:56.209 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:56.210 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.491733Z", 2022-01-11T14:34:56.210 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:56.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:56.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:56.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:56.211 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:34:56.211 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:56.211 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:56.211 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:34:56.212 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:56.212 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:56.212 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:56.212 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:34:56.213 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:56.213 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:56.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:56.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:56.213 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:34:56.214 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:56.214 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:56.214 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:56.215 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:56.215 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:56.215 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:56.216 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:56.216 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:56.216 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:56.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:56.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:34:56.217 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:34:56.217 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:34:56.217 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:34:56.218 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:34:56.218 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:34:56.218 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:34:56.218 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:34:56.219 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:34:56.219 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:34:56.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:56.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:56.219 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:34:56.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:34:56.220 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:34:56.220 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:34:56.220 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:34:56.221 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:34:56.221 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:34:56.221 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.491733_717.1 2022-01-11T14:35:01.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:01.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:01.224 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:35:01.331 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:35:01.331 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:35:01.642 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:35:01.643 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:35:01.643 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:35:01.643 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:35:01.644 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:35:01.644 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:35:01.644 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:01.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:01.644 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:35:01.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:01.645 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:35:01.645 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:01.645 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:01.646 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:35:01.646 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:01.646 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:35:01.647 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:01.647 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:35:01.647 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:01.647 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:01.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:01.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:01.648 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:35:01.648 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:01.648 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:01.649 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:35:01.649 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:01.649 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:01.649 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:01.650 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:35:01.650 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:01.650 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:01.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:01.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:01.651 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:35:01.651 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:01.652 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:01.652 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:01.652 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:01.652 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:01.652 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:01.653 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:01.653 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:01.653 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:01.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:01.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:01.654 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:35:01.654 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:01.654 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:01.654 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:01.655 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:01.655 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:01.655 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:01.655 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:01.655 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:01.656 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:01.656 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:01.656 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:35:01.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:01.657 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:35:01.657 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:35:01.657 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:35:01.657 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:01.657 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:35:01.659 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:35:06.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2022-01-11T14:35:06.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:06.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:06.663 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:35:06.873 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:35:06.873 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:35:06.922 INFO:teuthology.orchestra.run.smithi161.stderr:No --period given, using current period=86901227-b7e5-4fef-8f4e-54cce97cf2a2 2022-01-11T14:35:06.939 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2022-01-11T14:35:06.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:35:06.940 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641911394.664935_798.1", 2022-01-11T14:35:06.940 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:29:54.664935Z" 2022-01-11T14:35:06.941 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:35:06.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:35:06.941 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:35:06.941 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "2022-01-11T14:06:56.595927Z" 2022-01-11T14:35:06.942 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:35:06.942 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:35:06.943 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:35:06.943 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:35:06.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:35:06.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:35:06.944 INFO:teuthology.orchestra.run.smithi161.stdout: "marker": "", 2022-01-11T14:35:06.944 INFO:teuthology.orchestra.run.smithi161.stdout: "last_update": "0.000000" 2022-01-11T14:35:06.944 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:35:06.944 INFO:teuthology.orchestra.run.smithi161.stdout:] 2022-01-11T14:35:06.945 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2022-01-11T14:35:06.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:06.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:06.947 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:35:06.991 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:35:06.991 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:35:07.055 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:35:07.055 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:35:07.055 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:35:07.056 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:35:07.056 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:35:07.056 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:35:07.056 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:07.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:07.057 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:35:07.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:07.057 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:35:07.058 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:07.058 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:07.058 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:35:07.058 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:07.059 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:35:07.059 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:07.059 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:35:07.059 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:07.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:07.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:07.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:07.060 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:35:07.060 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:07.061 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:07.061 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:35:07.061 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:07.061 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:07.062 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:07.062 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:35:07.062 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:07.062 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:07.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:07.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:07.063 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:35:07.063 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:07.063 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:07.064 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:07.064 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:07.064 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:07.064 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:07.064 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:07.065 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:07.065 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:07.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:07.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:07.066 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:35:07.066 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:07.066 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:07.066 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:07.067 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:07.067 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:07.067 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:07.067 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:07.067 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:07.068 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:07.068 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:07.068 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:35:07.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:07.069 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:35:07.069 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:35:07.069 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:35:07.069 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:07.070 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:35:07.071 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:35:12.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:12.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:12.074 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:35:12.118 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:35:12.118 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:35:12.175 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:35:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:35:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:35:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:35:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:35:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:35:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:35:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:35:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.529400_722.1", 2022-01-11T14:35:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:35:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.529400Z", 2022-01-11T14:35:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:35:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:35:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:35:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:35:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:35:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:35:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:35:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:35:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:35:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:12.189 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:35:12.191 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.529400_722.1 2022-01-11T14:35:17.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:17.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:17.193 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:35:17.243 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:35:17.243 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:35:17.315 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:35:17.315 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:35:17.315 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:35:17.316 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:35:17.316 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:35:17.316 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:35:17.316 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:17.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:17.317 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:35:17.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:17.317 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:35:17.318 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:17.318 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:17.318 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:35:17.319 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:17.319 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:35:17.319 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:17.319 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:35:17.320 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:17.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:17.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:17.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:17.320 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:35:17.321 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:17.321 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:17.321 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:35:17.321 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:17.322 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:17.322 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:17.322 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:35:17.322 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:17.323 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:17.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:17.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:17.323 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:35:17.324 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:17.324 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:17.324 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:17.324 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:17.324 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:17.325 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:17.325 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:17.325 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:17.325 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:17.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:17.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:17.326 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:35:17.326 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:17.326 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:17.327 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:17.327 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:17.327 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:17.327 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:17.328 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:17.328 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:17.328 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:17.328 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:17.329 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:35:17.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:17.329 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:35:17.329 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:35:17.329 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:35:17.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:17.330 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:35:17.331 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:35:22.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:22.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:22.343 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:35:22.390 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:35:22.391 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:35:22.457 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:35:22.458 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:35:22.458 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:35:22.458 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:35:22.459 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:35:22.459 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:35:22.459 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:22.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:22.460 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:35:22.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:22.460 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:35:22.460 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:22.461 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:22.461 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:35:22.461 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:22.461 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:35:22.461 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:22.462 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:35:22.462 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:22.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:22.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:22.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:22.463 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:35:22.463 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:22.464 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:22.464 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:35:22.464 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:22.464 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:22.465 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:22.465 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:35:22.465 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:22.465 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:22.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:22.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:22.466 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:35:22.466 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:22.466 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:22.467 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:22.467 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:22.467 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:22.467 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:22.468 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:22.468 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:22.468 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:22.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:22.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:22.469 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:35:22.469 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:22.469 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:22.469 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:22.470 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:22.470 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:22.470 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:22.470 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:22.470 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:22.471 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:22.471 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:22.471 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:35:22.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:22.471 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:35:22.472 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:35:22.472 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:35:22.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:22.472 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:35:22.474 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:35:27.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:27.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:27.476 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:35:27.574 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:35:27.575 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:35:27.921 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:35:27.921 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:35:27.922 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:35:27.922 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:35:27.922 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:35:27.923 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:35:27.923 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:27.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:27.924 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:35:27.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:27.924 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:35:27.925 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:27.925 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:27.925 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:35:27.925 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:27.926 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:35:27.926 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:27.926 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:35:27.927 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:27.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:27.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:27.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:27.928 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:35:27.928 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:27.928 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:27.929 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:35:27.929 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:27.929 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:27.929 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:27.930 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:35:27.930 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:27.930 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:27.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:27.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:27.931 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:35:27.931 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:27.932 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:27.932 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:27.932 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:27.933 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:27.933 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:27.933 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:27.933 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:27.934 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:27.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:27.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:27.935 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:35:27.935 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:27.935 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:27.936 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:27.936 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:27.936 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:27.936 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:27.937 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:27.937 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:27.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:27.938 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:27.938 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:35:27.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:27.938 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:35:27.939 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:35:27.939 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:35:27.939 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:27.940 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:35:27.941 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:35:32.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:32.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:32.945 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:35:33.002 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:35:33.002 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:35:33.191 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:35:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:35:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:35:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:35:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:35:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:35:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:35:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:35:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.340992_709.1", 2022-01-11T14:35:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:35:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.340992Z", 2022-01-11T14:35:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:35:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:35:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:35:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:35:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:35:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:35:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:35:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:35:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:35:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:33.207 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:35:33.208 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.340992_709.1 2022-01-11T14:35:38.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:38.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:38.211 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:35:38.254 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:35:38.254 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:35:38.320 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:35:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:35:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:35:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:35:38.322 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:35:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:35:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:38.323 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:35:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:35:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:38.324 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.404090_712.1", 2022-01-11T14:35:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:35:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:38.325 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.404090Z", 2022-01-11T14:35:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:38.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:35:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:38.327 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:35:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:38.328 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:35:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:38.329 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:35:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:38.330 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:38.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:35:38.332 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:38.333 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:38.334 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:35:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:35:38.335 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:35:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:35:38.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:38.336 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:35:38.336 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.404090_712.1 2022-01-11T14:35:43.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:43.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:43.339 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:35:43.388 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:35:43.389 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:35:43.457 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:35:43.457 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:35:43.458 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:35:43.458 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:35:43.458 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:35:43.459 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:35:43.459 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:43.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:43.459 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:35:43.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:43.460 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:35:43.460 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:43.460 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:43.461 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.485421_716.1", 2022-01-11T14:35:43.461 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:43.461 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:35:43.461 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:43.462 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.485421Z", 2022-01-11T14:35:43.462 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:43.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:43.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:43.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:43.463 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:35:43.463 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:43.463 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:43.463 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:35:43.464 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:43.464 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:43.464 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:43.464 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:35:43.465 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:43.465 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:43.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:43.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:43.465 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:35:43.466 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:43.466 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:43.466 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:43.467 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:43.467 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:43.467 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:43.467 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:43.467 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:43.468 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:43.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:43.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:43.469 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:35:43.469 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:43.469 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:43.469 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:43.469 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:43.470 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:43.470 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:43.470 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:43.470 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:43.470 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:43.471 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:43.471 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:35:43.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:43.471 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:35:43.472 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:35:43.472 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:35:43.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:43.472 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:35:43.473 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.485421_716.1 2022-01-11T14:35:48.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:48.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:48.476 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:35:48.521 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:35:48.521 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:35:48.599 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:35:48.600 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:35:48.600 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:35:48.600 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:35:48.600 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:35:48.600 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:35:48.601 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:48.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:48.601 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:35:48.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:48.602 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:35:48.602 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:48.602 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:48.602 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:35:48.602 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:48.603 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:35:48.603 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:48.603 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:35:48.603 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:48.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:48.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:48.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:48.604 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:35:48.604 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:48.605 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:48.605 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:35:48.605 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:48.605 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:48.606 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:48.606 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:35:48.606 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:48.606 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:48.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:48.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:48.607 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:35:48.607 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:48.607 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:48.607 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:48.608 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:48.608 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:48.608 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:48.608 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:48.608 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:48.609 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:48.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:48.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:48.609 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:35:48.609 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:48.610 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:48.610 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:48.610 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:48.610 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:48.611 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:48.611 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:48.611 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:48.611 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:48.611 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:48.611 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:35:48.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:48.612 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:35:48.612 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:35:48.612 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:35:48.613 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:48.613 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:35:48.614 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:35:53.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:53.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:53.617 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:35:53.700 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:35:53.700 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:35:54.038 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:35:54.038 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:35:54.038 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:35:54.039 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:35:54.039 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:35:54.039 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:35:54.039 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:54.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:54.040 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:35:54.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:54.040 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:35:54.041 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:54.041 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:54.041 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:35:54.041 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:54.041 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:35:54.042 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:54.042 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:35:54.042 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:54.042 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:54.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:54.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:54.043 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:35:54.043 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:54.044 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:54.044 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:35:54.044 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:54.044 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:54.045 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:54.045 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:35:54.045 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:54.045 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:54.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:54.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:54.046 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:35:54.046 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:54.046 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:54.047 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:54.047 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:54.047 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:54.047 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:54.048 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:54.048 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:54.048 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:54.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:54.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:54.049 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:35:54.049 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:54.049 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:54.049 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:54.050 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:54.050 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:54.050 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:54.050 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:54.050 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:54.051 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:54.051 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:54.051 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:35:54.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:54.052 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:35:54.052 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:35:54.052 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:35:54.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:54.052 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:35:54.054 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:35:59.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:59.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:35:59.058 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:35:59.114 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:35:59.115 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:35:59.249 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:35:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:35:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:35:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:35:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:35:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:35:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:35:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:35:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:35:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:35:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:35:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:35:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:35:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:35:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:35:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:59.260 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:59.260 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:59.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:59.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:35:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:35:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:35:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:35:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:35:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:35:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:35:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:35:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:35:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:35:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:35:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:35:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:35:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:35:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:35:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:35:59.265 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:35:59.266 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:36:04.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:04.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:04.269 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:36:04.313 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:36:04.314 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:36:04.378 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:36:04.378 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:36:04.378 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:36:04.379 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:36:04.379 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:36:04.379 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:36:04.379 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:04.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:04.380 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:36:04.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:04.380 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:36:04.381 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:04.381 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:04.381 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:36:04.381 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:04.381 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:36:04.382 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:04.382 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:36:04.382 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:04.382 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:04.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:04.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:04.383 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:36:04.383 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:04.384 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:04.384 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:36:04.384 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:04.384 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:04.385 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:04.385 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:36:04.385 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:04.385 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:04.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:04.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:04.386 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:36:04.386 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:04.386 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:04.387 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:04.387 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:04.387 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:04.387 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:04.387 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:04.388 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:04.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:04.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:04.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:04.389 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:36:04.389 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:04.389 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:04.389 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:04.390 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:04.390 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:04.390 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:04.390 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:04.391 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:04.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:04.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:04.391 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:36:04.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:04.392 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:36:04.392 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:36:04.392 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:36:04.392 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:04.393 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:36:04.394 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:36:09.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:09.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:09.397 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:36:09.443 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:36:09.444 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:36:09.527 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:36:09.527 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:36:09.527 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:36:09.527 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:36:09.528 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:36:09.528 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:36:09.528 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:09.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:09.528 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:36:09.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:09.529 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:36:09.529 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:09.529 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:09.530 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910391.871317_735.1", 2022-01-11T14:36:09.530 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:09.530 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:36:09.530 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:09.530 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:11.871317Z", 2022-01-11T14:36:09.531 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:09.531 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:09.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:09.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:09.531 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:36:09.532 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:09.532 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:09.532 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:36:09.532 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:09.532 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:09.533 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:09.533 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:36:09.533 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:09.533 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:09.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:09.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:09.534 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:36:09.534 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:09.534 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:09.534 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:09.535 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:09.535 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:09.535 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:09.535 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:09.536 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:09.536 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:09.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:09.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:09.536 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:36:09.537 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:09.537 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:09.537 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:09.537 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:09.537 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:09.538 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:09.538 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:09.538 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:09.538 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:09.538 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:09.539 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:36:09.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:09.539 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:36:09.539 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:36:09.540 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:36:09.540 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:09.540 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:36:09.541 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910391.871317_735.1 2022-01-11T14:36:14.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:14.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:14.543 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:36:14.587 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:36:14.588 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:36:14.825 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:36:14.825 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:36:14.825 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:36:14.826 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:36:14.826 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:36:14.826 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:36:14.826 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:14.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:14.827 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:36:14.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:14.827 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:36:14.828 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:14.828 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:14.828 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:36:14.828 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:14.828 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:36:14.829 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:14.829 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:36:14.829 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:14.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:14.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:14.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:14.830 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:36:14.830 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:14.830 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:14.831 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:36:14.831 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:14.831 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:14.831 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:14.831 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:36:14.832 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:14.832 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:14.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:14.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:14.832 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:36:14.833 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:14.833 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:14.833 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:14.833 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:14.833 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:14.834 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:14.834 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:14.834 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:14.834 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:14.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:14.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:14.835 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:36:14.835 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:14.835 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:14.836 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:14.836 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:14.836 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:14.836 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:14.837 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:14.837 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:14.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:14.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:14.837 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:36:14.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:14.838 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:36:14.838 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:36:14.838 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:36:14.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:14.839 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:36:14.840 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:36:19.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:19.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:19.843 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:36:19.894 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:36:19.895 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:36:19.970 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:36:19.971 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:36:19.971 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:36:19.971 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:36:19.972 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:36:19.972 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:36:19.972 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:19.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:19.973 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:36:19.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:19.973 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:36:19.973 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:19.973 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:19.974 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.281170_707.1", 2022-01-11T14:36:19.974 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:19.974 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:36:19.974 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:19.975 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.281170Z", 2022-01-11T14:36:19.975 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:19.975 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:19.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:19.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:19.976 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:36:19.976 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:19.976 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:19.976 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:36:19.977 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:19.977 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:19.977 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:19.977 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:36:19.978 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:19.978 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:19.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:19.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:19.978 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:36:19.979 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:19.979 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:19.979 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:19.979 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:19.979 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:19.980 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:19.980 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:19.980 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:19.980 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:19.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:19.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:19.981 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:36:19.981 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:19.981 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:19.982 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:19.982 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:19.982 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:19.982 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:19.983 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:19.983 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:19.983 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:19.983 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:19.984 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:36:19.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:19.984 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:36:19.984 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:36:19.984 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:36:19.985 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:19.985 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:36:19.986 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.281170_707.1 2022-01-11T14:36:24.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:24.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:24.990 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:36:25.039 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:36:25.039 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:36:25.095 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:36:25.095 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:36:25.095 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:36:25.096 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:36:25.096 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:36:25.096 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:36:25.096 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:25.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:25.097 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:36:25.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:25.097 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:36:25.097 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:25.098 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:25.098 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:36:25.098 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:25.098 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:36:25.098 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:36:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:25.100 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:36:25.100 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:25.100 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:25.100 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:36:25.100 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:25.101 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:25.101 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:25.101 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:36:25.102 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:25.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:25.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:25.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:25.103 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:36:25.103 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:25.103 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:25.104 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:25.104 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:25.104 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:25.104 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:25.104 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:25.105 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:25.105 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:25.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:25.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:25.106 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:36:25.106 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:25.106 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:25.107 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:25.107 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:25.107 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:25.107 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:25.108 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:25.108 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:25.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:25.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:25.108 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:36:25.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:25.109 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:36:25.109 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:36:25.109 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:36:25.110 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:25.110 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:36:25.111 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:36:30.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:30.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:30.113 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:36:30.156 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:36:30.156 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:36:30.221 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:36:30.222 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:36:30.222 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:36:30.223 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:36:30.223 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:36:30.224 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:36:30.224 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:30.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:30.225 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:36:30.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:30.225 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:36:30.226 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:30.226 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:30.227 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.257680_704.1", 2022-01-11T14:36:30.227 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:30.228 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:36:30.228 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:30.228 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.257680Z", 2022-01-11T14:36:30.229 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:30.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:30.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:30.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:30.230 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:36:30.231 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:30.231 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:30.232 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:36:30.232 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:30.233 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:30.233 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:30.233 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:36:30.234 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:30.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:30.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:30.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:30.236 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:36:30.236 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:30.237 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:30.237 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:30.237 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:30.237 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:30.238 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:30.238 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:30.238 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:30.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:30.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:30.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:30.239 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:36:30.239 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:30.239 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:30.240 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:30.240 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:30.240 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:30.240 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:30.241 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:30.241 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:30.241 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:30.241 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:30.241 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:36:30.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:30.242 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:36:30.242 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:36:30.242 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:36:30.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:30.243 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:36:30.244 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.257680_704.1 2022-01-11T14:36:35.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:35.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:35.247 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:36:35.294 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:36:35.294 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:36:35.353 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:36:35.353 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:36:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:36:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:36:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:36:35.354 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:36:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:35.355 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:36:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:36:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:35.356 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.519237_721.1", 2022-01-11T14:36:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:36:35.357 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.519237Z", 2022-01-11T14:36:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:35.358 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:36:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:36:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:35.360 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:36:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:36:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:35.362 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:35.363 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:35.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:36:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:35.365 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:35.366 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:36:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:35.367 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:36:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:36:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:36:35.368 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:35.368 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:36:35.369 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.519237_721.1 2022-01-11T14:36:40.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:40.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:40.373 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:36:40.435 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:36:40.436 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:36:40.751 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:36:40.751 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:36:40.751 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:36:40.752 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:36:40.752 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:36:40.752 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:36:40.753 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:40.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:40.753 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:36:40.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:40.753 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:36:40.754 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:40.754 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:40.754 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910391.801637_734.1", 2022-01-11T14:36:40.754 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:40.755 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:36:40.755 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:40.755 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:11.801637Z", 2022-01-11T14:36:40.755 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:40.756 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:40.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:40.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:40.756 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:36:40.757 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:40.757 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:40.757 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:36:40.757 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:40.758 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:40.758 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:40.758 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:36:40.758 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:40.758 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:40.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:40.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:40.759 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:36:40.759 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:40.760 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:40.760 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:40.760 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:40.760 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:40.760 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:40.761 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:40.761 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:40.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:40.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:40.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:40.762 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:36:40.762 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:40.762 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:40.763 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:40.763 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:40.763 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:40.763 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:40.764 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:40.764 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:40.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:40.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:40.765 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:36:40.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:40.765 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:36:40.765 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:36:40.765 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:36:40.766 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:40.766 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:36:40.767 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910391.801637_734.1 2022-01-11T14:36:45.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:45.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:45.770 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:36:45.820 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:36:45.820 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:36:45.995 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:36:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:36:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:36:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:36:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:36:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:36:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:36:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:36:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.285437_708.1", 2022-01-11T14:36:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:36:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.285437Z", 2022-01-11T14:36:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:36:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:36:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:36:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:36:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:36:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:36:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:36:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:36:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:36:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:46.010 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:36:46.011 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.285437_708.1 2022-01-11T14:36:51.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:51.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:51.014 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:36:51.057 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:36:51.058 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:36:51.126 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:36:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:36:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:36:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:36:51.127 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:36:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:36:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:51.128 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:36:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:36:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:51.129 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:36:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:36:51.130 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:36:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:51.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:36:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:51.132 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:36:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:51.133 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:36:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:51.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:36:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:51.135 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:51.136 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:51.137 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:36:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:51.138 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:51.139 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:36:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:51.140 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:36:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:36:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:36:51.141 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:51.141 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:36:51.143 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:36:56.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:56.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:36:56.145 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:36:56.317 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:36:56.318 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:36:56.383 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:36:56.384 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:36:56.384 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:36:56.384 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:36:56.385 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:36:56.385 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:36:56.385 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:56.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:56.386 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:36:56.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:56.386 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:36:56.386 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:56.387 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:56.387 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.482535_715.1", 2022-01-11T14:36:56.387 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:56.387 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:36:56.387 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:56.388 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.482535Z", 2022-01-11T14:36:56.388 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:56.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:56.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:56.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:56.389 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:36:56.389 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:56.390 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:56.390 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:36:56.390 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:56.390 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:56.391 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:56.391 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:36:56.391 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:56.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:56.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:56.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:56.392 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:36:56.392 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:56.393 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:56.393 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:56.393 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:56.393 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:56.394 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:56.394 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:56.394 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:56.394 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:56.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:56.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:36:56.395 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:36:56.395 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:36:56.395 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:36:56.396 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:36:56.396 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:36:56.396 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:36:56.396 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:36:56.397 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:36:56.397 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:36:56.397 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:56.397 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:56.398 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:36:56.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:36:56.398 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:36:56.398 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:36:56.399 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:36:56.399 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:36:56.399 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:36:56.399 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.482535_715.1 2022-01-11T14:37:01.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:01.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:01.402 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:37:01.446 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:37:01.447 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:37:01.507 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:37:01.508 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:37:01.508 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:37:01.508 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:37:01.508 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:37:01.509 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:37:01.509 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:01.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:01.509 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:37:01.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:01.510 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:37:01.510 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:01.511 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:01.511 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.404090_712.1", 2022-01-11T14:37:01.511 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:01.511 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:37:01.512 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:01.512 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.404090Z", 2022-01-11T14:37:01.512 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:01.512 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:01.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:01.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:01.513 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:37:01.513 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:01.514 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:01.514 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:37:01.515 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:01.515 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:01.515 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:01.515 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:37:01.515 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:01.516 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:01.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:01.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:01.516 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:37:01.517 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:01.517 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:01.517 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:01.517 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:01.518 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:01.518 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:01.518 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:01.518 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:01.518 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:01.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:01.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:01.519 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:37:01.519 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:01.520 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:01.520 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:01.520 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:01.520 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:01.521 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:01.522 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:01.522 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:01.522 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:01.522 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:01.523 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:37:01.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:01.523 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:37:01.523 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:37:01.524 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:37:01.524 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:01.524 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:37:01.526 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.404090_712.1 2022-01-11T14:37:06.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:06.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:06.529 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:37:06.603 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:37:06.604 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:37:06.912 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:37:06.912 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:37:06.913 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:37:06.913 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:37:06.913 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:37:06.914 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:37:06.914 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:06.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:06.914 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:37:06.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:06.915 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:37:06.915 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:06.915 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:06.916 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.344534_710.1", 2022-01-11T14:37:06.916 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:06.916 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:37:06.916 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:06.917 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.344534Z", 2022-01-11T14:37:06.917 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:06.917 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:06.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:06.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:06.918 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:37:06.918 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:06.918 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:06.919 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:37:06.919 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:06.919 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:06.919 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:06.920 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:37:06.920 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:06.920 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:06.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:06.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:06.921 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:37:06.921 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:06.921 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:06.921 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:06.922 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:06.922 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:06.922 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:06.922 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:06.922 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:06.923 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:06.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:06.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:06.923 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:37:06.924 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:06.924 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:06.924 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:06.924 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:06.925 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:06.925 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:06.925 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:06.925 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:06.925 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:06.926 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:06.926 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:37:06.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:06.926 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:37:06.927 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:37:06.927 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:37:06.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:06.927 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:37:06.929 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.344534_710.1 2022-01-11T14:37:11.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:11.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:11.932 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:37:11.997 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:37:11.997 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:37:12.188 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:37:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:37:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:37:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:37:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:37:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:37:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:37:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:37:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.281170_707.1", 2022-01-11T14:37:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:37:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.281170Z", 2022-01-11T14:37:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:37:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:37:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:37:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:37:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:37:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:37:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:37:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:37:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:37:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:12.204 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:37:12.206 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.281170_707.1 2022-01-11T14:37:17.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:17.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:17.208 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:37:17.252 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:37:17.252 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:37:17.333 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:37:17.334 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:37:17.334 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:37:17.335 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:37:17.335 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:37:17.335 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:37:17.335 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:17.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:17.336 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:37:17.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:17.336 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:37:17.337 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:17.337 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:17.337 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.285437_708.1", 2022-01-11T14:37:17.337 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:17.337 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:37:17.338 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:17.338 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.285437Z", 2022-01-11T14:37:17.338 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:17.338 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:17.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:17.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:17.339 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:37:17.339 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:17.340 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:17.340 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:37:17.340 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:17.340 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:17.340 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:17.341 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:37:17.341 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:17.341 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:17.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:17.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:17.342 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:37:17.342 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:17.342 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:17.343 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:17.343 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:17.343 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:17.343 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:17.343 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:17.344 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:17.344 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:17.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:17.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:17.345 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:37:17.345 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:17.345 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:17.346 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:17.346 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:17.346 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:17.347 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:17.347 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:17.347 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:17.347 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:17.348 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:17.348 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:37:17.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:17.348 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:37:17.349 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:37:17.349 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:37:17.349 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:17.350 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:37:17.351 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.285437_708.1 2022-01-11T14:37:22.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:22.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:22.354 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:37:22.403 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:37:22.404 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:37:22.466 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:37:22.467 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:37:22.467 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:37:22.467 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:37:22.468 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:37:22.468 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:37:22.468 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:22.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:22.469 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:37:22.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:22.469 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:37:22.469 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:22.469 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:22.470 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:37:22.470 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:22.470 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:37:22.470 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:22.471 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:37:22.471 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:22.471 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:22.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:22.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:22.472 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:37:22.472 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:22.472 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:22.473 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:37:22.473 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:22.473 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:22.473 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:22.473 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:37:22.474 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:22.474 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:22.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:22.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:22.475 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:37:22.475 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:22.475 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:22.475 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:22.475 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:22.476 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:22.476 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:22.476 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:22.476 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:22.477 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:22.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:22.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:22.478 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:37:22.478 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:22.478 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:22.478 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:22.478 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:22.479 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:22.479 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:22.479 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:22.479 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:22.480 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:22.480 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:22.480 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:37:22.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:22.481 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:37:22.481 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:37:22.481 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:37:22.481 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:22.481 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:37:22.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:37:27.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:27.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:27.485 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:37:27.534 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:37:27.535 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:37:27.619 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:37:27.620 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:37:27.620 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:37:27.620 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:37:27.621 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:37:27.621 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:37:27.621 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:27.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:27.622 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:37:27.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:27.622 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:37:27.622 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:27.623 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:27.623 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:37:27.623 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:27.623 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:37:27.623 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:27.624 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:37:27.624 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:27.624 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:27.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:27.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:27.625 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:37:27.625 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:27.625 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:27.626 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:37:27.626 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:27.626 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:27.626 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:27.627 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:37:27.627 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:27.627 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:27.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:27.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:27.628 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:37:27.628 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:27.628 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:27.628 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:27.629 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:27.629 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:27.629 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:27.629 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:27.630 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:27.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:27.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:27.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:27.630 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:37:27.631 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:27.631 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:27.631 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:27.631 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:27.632 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:27.632 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:27.632 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:27.632 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:27.633 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:27.633 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:27.633 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:37:27.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:27.633 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:37:27.634 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:37:27.634 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:37:27.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:27.634 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:37:27.636 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:37:32.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:32.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:32.638 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:37:32.708 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:37:32.708 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:37:33.086 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:37:33.086 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:37:33.086 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:37:33.087 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:37:33.087 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:37:33.087 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:37:33.087 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:33.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:33.088 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:37:33.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:33.088 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:37:33.089 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:33.089 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:33.089 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.344534_710.1", 2022-01-11T14:37:33.089 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:33.090 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:37:33.090 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:33.090 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.344534Z", 2022-01-11T14:37:33.090 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:33.090 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:33.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:33.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:33.091 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:37:33.091 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:33.092 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:33.092 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:37:33.092 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:33.092 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:33.093 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:33.093 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:37:33.093 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:33.093 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:33.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:33.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:33.094 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:37:33.094 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:33.094 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:33.095 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:33.095 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:33.095 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:33.095 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:33.096 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:33.096 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:33.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:33.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:33.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:33.097 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:37:33.097 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:33.097 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:33.098 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:33.098 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:33.098 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:33.098 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:33.099 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:33.099 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:33.099 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:33.099 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:33.100 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:37:33.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:33.100 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:37:33.100 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:37:33.100 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:37:33.101 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:33.101 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:37:33.102 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.344534_710.1 2022-01-11T14:37:38.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:38.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:38.105 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:37:38.170 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:37:38.170 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:37:38.353 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:37:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:37:38.354 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:37:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:37:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:37:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:37:38.355 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:37:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:38.356 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:37:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.476921_714.1", 2022-01-11T14:37:38.357 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:37:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.476921Z", 2022-01-11T14:37:38.358 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:38.359 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:37:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:38.360 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:37:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:38.361 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:37:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:38.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:37:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:38.363 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:38.364 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:37:38.365 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:38.366 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:38.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:37:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:38.368 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:37:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:37:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:37:38.369 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:38.369 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:37:38.371 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.476921_714.1 2022-01-11T14:37:43.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:43.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:43.373 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:37:43.416 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:37:43.417 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:37:43.469 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:37:43.469 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:37:43.470 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:37:43.470 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:37:43.470 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:37:43.471 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:37:43.471 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:43.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:43.471 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:37:43.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:43.472 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:37:43.472 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:43.472 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:43.473 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.404090_712.1", 2022-01-11T14:37:43.473 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:43.473 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:37:43.473 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:43.474 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.404090Z", 2022-01-11T14:37:43.474 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:43.474 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:43.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:43.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:43.475 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:37:43.475 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:43.475 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:43.476 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:37:43.476 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:43.476 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:43.476 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:43.477 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:37:43.477 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:43.477 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:43.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:43.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:43.478 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:37:43.479 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:43.479 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:43.479 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:43.479 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:43.479 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:43.480 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:43.480 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:43.481 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:43.481 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:43.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:43.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:43.482 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:37:43.482 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:43.482 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:43.482 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:43.482 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:43.483 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:43.483 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:43.483 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:43.483 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:43.484 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:43.484 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:43.485 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:37:43.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:43.485 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:37:43.485 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:37:43.486 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:37:43.486 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:43.486 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:37:43.487 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.404090_712.1 2022-01-11T14:37:48.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:48.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:48.491 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:37:48.541 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:37:48.541 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:37:48.619 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:37:48.620 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:37:48.620 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:37:48.620 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:37:48.621 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:37:48.621 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:37:48.621 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:48.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:48.621 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:37:48.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:48.622 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:37:48.622 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:48.622 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:48.623 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:37:48.623 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:48.623 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:37:48.623 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:48.624 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:37:48.624 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:48.624 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:48.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:48.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:48.625 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:37:48.625 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:48.625 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:48.625 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:37:48.626 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:48.626 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:48.626 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:48.626 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:37:48.627 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:48.627 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:48.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:48.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:48.627 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:37:48.628 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:48.628 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:48.628 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:48.628 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:48.629 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:48.629 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:48.629 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:48.629 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:48.629 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:48.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:48.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:48.630 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:37:48.630 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:48.631 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:48.631 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:48.631 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:48.631 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:48.632 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:48.632 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:48.632 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:48.632 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:48.632 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:48.633 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:37:48.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:48.633 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:37:48.633 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:37:48.633 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:37:48.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:48.634 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:37:48.635 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:37:53.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:53.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:53.639 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:37:53.686 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:37:53.686 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:37:53.750 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:37:53.750 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:37:53.750 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:37:53.751 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:37:53.751 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:37:53.751 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:37:53.751 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:53.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:53.752 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:37:53.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:53.752 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:37:53.753 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:53.753 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:53.753 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.261451_706.1", 2022-01-11T14:37:53.753 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:53.754 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:37:53.754 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:53.754 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.261451Z", 2022-01-11T14:37:53.754 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:53.754 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:53.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:53.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:53.755 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:37:53.755 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:53.756 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:53.756 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:37:53.756 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:53.756 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:53.757 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:53.757 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:37:53.757 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:53.757 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:53.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:53.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:53.758 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:37:53.758 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:53.759 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:53.759 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:53.759 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:53.759 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:53.760 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:53.760 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:53.760 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:53.760 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:53.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:53.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:53.761 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:37:53.761 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:53.761 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:53.762 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:53.762 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:53.762 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:53.762 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:53.763 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:53.763 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:53.763 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:53.763 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:53.764 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:37:53.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:53.764 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:37:53.764 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:37:53.764 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:37:53.765 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:53.765 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:37:53.766 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.261451_706.1 2022-01-11T14:37:58.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:58.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:37:58.769 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:37:58.830 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:37:58.831 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:37:59.203 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:37:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:37:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:37:59.204 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:37:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:37:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:37:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:59.205 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:37:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:37:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:59.206 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.257680_704.1", 2022-01-11T14:37:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:37:59.207 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.257680Z", 2022-01-11T14:37:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:59.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:37:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:59.209 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:37:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:59.210 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:37:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:59.211 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:37:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:59.212 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:59.213 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:37:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:37:59.214 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:37:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:37:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:37:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:37:59.215 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:37:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:37:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:37:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:37:59.216 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:37:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:37:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:37:59.217 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:37:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:37:59.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:37:59.218 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:37:59.219 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.257680_704.1 2022-01-11T14:38:04.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:04.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:04.221 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:38:04.277 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:38:04.278 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:38:04.411 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:38:04.411 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:38:04.411 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:38:04.412 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:38:04.412 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:38:04.412 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:38:04.412 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:04.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:04.413 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:38:04.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:04.413 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:38:04.413 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:04.414 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:04.414 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:38:04.414 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:04.414 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:38:04.415 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:04.415 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:38:04.415 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:04.415 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:04.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:04.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:04.416 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:38:04.416 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:04.416 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:04.417 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:38:04.417 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:04.417 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:04.417 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:04.417 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:38:04.418 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:04.418 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:04.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:04.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:04.420 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:38:04.420 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:04.420 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:04.420 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:04.421 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:04.421 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:04.421 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:04.421 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:04.421 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:04.422 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:04.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:04.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:04.422 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:38:04.423 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:04.423 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:04.423 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:04.423 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:04.423 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:04.424 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:04.424 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:04.424 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:04.424 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:04.425 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:04.425 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:38:04.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:04.425 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:38:04.425 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:38:04.426 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:38:04.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:04.426 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:38:04.427 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:38:09.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:09.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:09.430 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:38:09.473 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:38:09.474 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:38:09.556 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:38:09.557 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:38:09.557 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:38:09.557 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:38:09.557 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:38:09.558 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:38:09.558 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:09.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:09.558 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:38:09.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:09.559 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:38:09.559 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:09.559 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:09.560 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:38:09.560 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:09.560 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:38:09.560 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:09.561 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:38:09.561 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:09.561 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:09.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:09.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:09.562 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:38:09.562 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:09.562 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:09.562 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:38:09.563 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:09.563 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:09.563 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:09.563 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:38:09.564 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:09.564 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:09.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:09.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:09.564 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:38:09.565 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:09.565 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:09.565 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:09.565 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:09.566 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:09.566 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:09.566 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:09.566 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:09.566 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:09.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:09.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:09.567 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:38:09.567 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:09.568 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:09.568 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:09.568 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:09.568 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:09.568 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:09.569 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:09.569 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:09.569 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:09.569 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:09.570 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:38:09.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:09.570 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:38:09.570 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:38:09.571 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:38:09.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:09.571 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:38:09.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:38:14.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:14.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:14.575 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:38:14.624 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:38:14.625 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:38:14.696 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:38:14.697 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:38:14.697 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:38:14.697 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:38:14.697 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:38:14.697 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:38:14.698 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:14.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:14.698 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:38:14.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:14.699 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:38:14.699 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:14.699 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:14.699 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.404090_712.1", 2022-01-11T14:38:14.700 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:14.700 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:38:14.700 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:14.700 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.404090Z", 2022-01-11T14:38:14.701 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:14.701 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:14.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:14.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:14.702 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:38:14.702 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:14.702 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:14.702 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:38:14.702 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:14.703 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:14.703 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:14.703 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:38:14.703 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:14.704 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:14.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:14.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:14.704 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:38:14.705 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:14.705 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:14.705 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:14.705 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:14.705 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:14.706 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:14.706 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:14.706 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:14.706 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:14.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:14.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:14.707 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:38:14.707 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:14.708 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:14.708 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:14.708 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:14.708 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:14.708 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:14.709 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:14.709 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:14.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:14.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:14.710 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:38:14.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:14.710 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:38:14.710 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:38:14.710 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:38:14.711 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:14.711 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:38:14.712 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.404090_712.1 2022-01-11T14:38:19.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:19.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:19.715 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:38:19.765 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:38:19.765 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:38:19.832 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:38:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:38:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:38:19.833 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:38:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:38:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:38:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:19.834 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:38:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:38:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:19.835 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.404090_712.1", 2022-01-11T14:38:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:38:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:19.836 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.404090Z", 2022-01-11T14:38:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:19.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:38:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:38:19.838 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:38:19.839 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:38:19.840 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:19.841 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:19.842 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:38:19.843 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:19.844 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:19.845 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:19.846 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:38:19.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:19.846 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:38:19.846 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:38:19.847 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:38:19.847 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:19.847 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:38:19.848 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.404090_712.1 2022-01-11T14:38:24.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:24.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:24.851 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:38:24.908 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:38:24.908 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:38:25.295 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:38:25.296 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:38:25.296 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:38:25.296 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:38:25.296 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:38:25.296 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:38:25.297 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:25.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:25.297 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:38:25.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:25.298 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:38:25.298 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:25.298 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:25.298 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.281170_707.1", 2022-01-11T14:38:25.299 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:25.299 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:38:25.299 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:25.299 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.281170Z", 2022-01-11T14:38:25.299 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:25.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:25.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:25.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:25.300 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:38:25.301 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:25.301 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:25.301 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:38:25.301 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:25.301 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:25.302 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:25.302 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:38:25.302 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:25.302 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:25.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:25.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:25.303 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:38:25.303 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:25.304 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:25.304 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:25.304 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:25.304 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:25.304 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:25.305 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:25.305 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:25.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:25.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:25.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:25.306 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:38:25.306 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:25.306 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:25.306 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:25.307 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:25.307 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:25.307 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:25.307 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:25.308 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:25.308 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:25.308 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:25.308 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:38:25.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:25.309 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:38:25.309 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:38:25.309 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:38:25.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:25.310 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:38:25.310 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.281170_707.1 2022-01-11T14:38:30.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:30.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:30.313 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:38:30.379 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:38:30.379 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:38:30.577 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:38:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:38:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:38:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:38:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:38:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:38:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:30.580 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:38:30.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:30.580 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:38:30.580 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.491733_717.1", 2022-01-11T14:38:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:38:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.491733Z", 2022-01-11T14:38:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:38:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:38:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:38:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:38:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:38:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:38:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:38:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:38:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:38:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:30.592 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:38:30.594 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.491733_717.1 2022-01-11T14:38:35.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:35.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:35.597 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:38:35.645 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:38:35.645 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:38:35.728 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:38:35.729 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:38:35.729 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:38:35.729 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:38:35.729 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:38:35.730 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:38:35.730 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:35.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:35.731 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:38:35.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:35.732 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:38:35.732 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:35.732 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:35.732 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.344534_710.1", 2022-01-11T14:38:35.732 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:35.733 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:38:35.733 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:35.733 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.344534Z", 2022-01-11T14:38:35.733 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:35.734 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:35.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:35.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:35.734 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:38:35.734 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:35.735 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:35.735 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:38:35.735 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:35.736 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:35.736 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:35.736 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:38:35.736 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:35.737 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:35.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:35.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:35.737 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:38:35.737 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:35.738 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:35.738 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:35.739 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:35.739 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:35.740 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:35.740 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:35.741 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:35.741 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:35.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:35.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:35.742 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:38:35.742 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:35.742 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:35.742 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:38:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:38:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:38:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:38:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:35.746 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:38:35.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.344534_710.1 2022-01-11T14:38:40.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:40.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:40.749 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:38:40.794 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:38:40.795 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:38:40.875 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:38:40.875 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:38:40.876 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:38:40.876 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:38:40.876 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:38:40.876 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:38:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:38:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:38:40.878 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:40.878 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.257680_704.1", 2022-01-11T14:38:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:38:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.257680Z", 2022-01-11T14:38:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:38:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:38:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:38:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:38:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:38:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:38:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:38:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:38:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:38:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:40.891 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:38:40.892 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.257680_704.1 2022-01-11T14:38:45.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:45.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:45.894 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:38:45.939 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:38:45.940 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:38:46.017 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:38:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:38:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:38:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:38:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:38:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:38:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:38:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:38:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.257680_704.1", 2022-01-11T14:38:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:38:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.257680Z", 2022-01-11T14:38:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:38:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:38:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:38:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:38:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:38:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:38:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:38:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:38:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:38:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:46.033 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:38:46.034 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.257680_704.1 2022-01-11T14:38:51.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:51.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:51.037 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:38:51.136 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:38:51.137 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:38:51.427 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:38:51.428 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:38:51.428 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:38:51.428 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:38:51.428 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:38:51.429 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:38:51.429 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:51.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:51.430 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:38:51.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:51.430 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:38:51.430 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:51.431 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:51.431 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:38:51.431 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:51.431 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:38:51.431 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:51.432 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:38:51.432 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:51.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:51.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:51.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:51.433 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:38:51.433 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:51.433 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:51.433 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:38:51.434 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:51.434 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:51.434 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:51.435 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:38:51.435 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:51.435 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:51.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:51.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:51.436 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:38:51.437 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:51.437 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:51.437 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:51.437 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:51.438 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:51.438 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:51.438 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:51.438 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:51.438 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:51.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:51.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:51.439 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:38:51.439 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:51.439 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:51.440 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:51.440 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:51.440 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:51.440 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:51.441 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:51.442 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:51.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:51.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:51.442 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:38:51.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:51.443 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:38:51.443 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:38:51.443 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:38:51.444 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:51.444 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:38:51.445 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:38:56.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:56.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:38:56.448 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:38:56.504 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:38:56.504 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:38:56.728 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:38:56.729 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:38:56.729 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:38:56.730 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:38:56.730 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:38:56.731 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:38:56.731 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:56.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:56.732 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:38:56.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:56.733 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:38:56.733 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:56.733 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:56.734 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.340992_709.1", 2022-01-11T14:38:56.734 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:56.735 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:38:56.735 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:56.736 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.340992Z", 2022-01-11T14:38:56.736 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:56.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:56.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:56.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:56.738 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:38:56.738 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:56.739 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:56.739 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:38:56.739 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:56.740 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:56.740 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:56.741 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:38:56.741 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:56.742 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:56.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:56.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:56.743 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:38:56.743 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:56.744 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:56.744 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:56.745 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:56.745 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:56.746 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:56.746 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:56.746 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:56.747 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:56.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:56.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:38:56.748 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:38:56.749 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:38:56.749 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:38:56.750 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:38:56.750 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:38:56.751 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:38:56.751 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:38:56.751 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:38:56.752 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:38:56.752 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:56.753 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:56.753 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:38:56.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:38:56.754 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:38:56.755 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:38:56.755 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:38:56.756 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:38:56.756 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:38:56.758 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.340992_709.1 2022-01-11T14:39:01.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:01.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:01.762 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:39:01.812 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:39:01.812 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:39:01.885 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:39:01.885 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:39:01.886 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:39:01.886 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:39:01.887 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:39:01.887 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:39:01.888 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:01.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:01.889 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:39:01.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:01.890 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:39:01.890 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:01.891 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:01.891 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.404090_712.1", 2022-01-11T14:39:01.891 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:01.892 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:39:01.893 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:01.893 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.404090Z", 2022-01-11T14:39:01.894 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:01.894 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:01.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:01.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:01.896 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:39:01.896 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:01.896 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:01.897 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:39:01.897 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:01.898 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:01.898 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:01.899 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:39:01.899 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:01.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:01.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:01.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:01.901 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:39:01.901 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:01.902 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:01.902 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:01.903 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:01.903 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:01.904 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:01.904 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:01.905 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:01.905 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:01.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:01.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:01.906 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:39:01.906 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:01.906 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:01.906 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:01.907 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:01.907 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:01.907 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:01.907 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:01.908 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:01.908 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:01.908 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:01.908 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:39:01.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:01.909 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:39:01.909 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:39:01.909 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:39:01.909 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:01.910 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:39:01.911 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.404090_712.1 2022-01-11T14:39:06.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:06.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:06.915 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:39:06.965 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:39:06.965 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:39:07.033 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:39:07.033 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:39:07.034 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:39:07.034 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:39:07.035 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:39:07.035 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:39:07.035 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:07.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:07.035 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:39:07.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:07.036 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:39:07.036 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:07.036 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:07.037 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:39:07.037 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:07.037 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:39:07.037 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:07.038 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:39:07.038 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:07.038 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:07.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:07.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:07.039 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:39:07.039 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:07.039 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:07.039 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:39:07.040 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:07.040 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:07.040 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:07.040 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:39:07.041 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:07.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:07.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:07.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:07.041 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:39:07.042 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:07.042 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:07.042 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:07.042 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:07.043 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:07.043 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:07.043 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:07.043 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:07.044 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:07.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:07.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:07.044 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:39:07.045 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:07.045 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:07.045 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:07.045 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:07.046 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:07.046 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:07.046 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:07.046 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:07.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:07.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:07.047 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:39:07.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:07.047 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:39:07.048 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:39:07.048 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:39:07.048 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:07.048 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:39:07.050 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:39:12.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:12.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:12.054 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:39:12.104 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:39:12.105 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:39:12.175 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:39:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:39:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:39:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:39:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:39:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:39:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:39:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:39:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.340992_709.1", 2022-01-11T14:39:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:39:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.340992Z", 2022-01-11T14:39:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:39:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:39:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:39:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:39:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:39:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:39:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:39:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:39:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:39:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:12.198 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:39:12.199 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.340992_709.1 2022-01-11T14:39:17.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:17.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:17.202 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:39:17.277 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:39:17.277 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:39:17.672 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:39:17.673 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:39:17.673 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:39:17.673 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:39:17.673 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:39:17.674 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:39:17.674 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:17.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:17.674 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:39:17.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:17.675 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:39:17.675 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:17.675 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:17.675 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.485421_716.1", 2022-01-11T14:39:17.676 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:17.676 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:39:17.676 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:17.676 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.485421Z", 2022-01-11T14:39:17.677 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:17.677 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:17.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:17.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:17.677 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:39:17.678 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:17.678 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:17.678 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:39:17.679 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:17.679 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:17.679 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:17.679 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:39:17.679 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:17.680 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:17.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:17.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:17.680 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:39:17.681 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:17.681 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:17.681 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:17.681 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:17.682 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:17.682 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:17.682 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:17.682 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:17.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:17.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:17.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:17.683 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:39:17.684 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:17.684 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:17.684 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:17.684 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:17.684 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:17.685 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:17.685 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:17.685 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:17.686 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:17.686 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:17.686 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:39:17.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:17.686 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:39:17.687 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:39:17.687 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:39:17.687 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:17.687 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:39:17.689 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.485421_716.1 2022-01-11T14:39:22.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:22.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:22.691 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:39:22.736 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:39:22.736 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:39:22.872 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:39:22.873 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:39:22.873 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:39:22.874 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:39:22.874 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:39:22.875 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:39:22.875 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:22.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:22.876 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:39:22.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:22.876 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:39:22.876 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:22.877 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:22.877 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.257680_704.1", 2022-01-11T14:39:22.877 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:22.877 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:39:22.878 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:22.878 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.257680Z", 2022-01-11T14:39:22.878 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:22.878 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:22.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:22.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:22.879 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:39:22.879 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:22.879 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:22.880 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:39:22.880 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:22.880 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:22.880 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:22.881 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:39:22.881 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:22.881 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:22.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:22.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:22.882 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:39:22.883 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:22.883 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:22.884 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:22.884 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:22.885 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:22.885 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:22.885 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:22.886 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:22.886 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:22.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:22.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:22.888 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:39:22.888 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:22.888 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:22.889 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:22.889 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:22.890 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:22.890 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:22.891 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:22.891 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:22.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:22.892 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:22.892 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:39:22.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:22.893 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:39:22.894 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:39:22.894 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:39:22.895 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:22.895 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:39:22.896 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.257680_704.1 2022-01-11T14:39:27.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:27.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:27.899 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:39:27.957 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:39:27.958 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:39:28.025 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:39:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:39:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:39:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:39:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:39:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:39:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:39:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:39:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.517775_720.1", 2022-01-11T14:39:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:39:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.517775Z", 2022-01-11T14:39:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:39:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:39:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:39:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:39:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:39:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:39:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:39:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:39:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:39:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:28.040 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:39:28.042 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.517775_720.1 2022-01-11T14:39:33.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:33.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:33.045 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:39:33.088 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:39:33.088 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:39:33.154 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:39:33.155 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:39:33.155 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:39:33.155 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:39:33.155 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:39:33.156 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:39:33.156 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:33.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:33.156 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:39:33.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:33.157 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:39:33.157 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:33.157 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:33.157 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.340992_709.1", 2022-01-11T14:39:33.158 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:33.158 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:39:33.158 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:33.158 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.340992Z", 2022-01-11T14:39:33.159 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:33.159 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:33.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:33.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:33.160 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:39:33.160 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:33.160 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:33.160 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:39:33.160 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:39:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:39:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:39:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:39:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:39:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:39:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:39:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:33.171 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:39:33.173 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.340992_709.1 2022-01-11T14:39:38.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:38.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:38.176 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:39:38.226 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:39:38.226 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:39:38.304 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:39:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:39:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:39:38.305 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:39:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:39:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:39:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:38.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:39:38.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:39:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.281170_707.1", 2022-01-11T14:39:38.308 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:39:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:38.309 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.281170Z", 2022-01-11T14:39:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:38.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:39:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:38.311 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:39:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:38.312 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:39:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:38.313 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:39:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:38.314 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:38.315 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:38.316 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:39:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:38.317 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:38.318 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:39:38.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:38.320 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:39:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:39:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:39:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:38.321 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:39:38.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.281170_707.1 2022-01-11T14:39:43.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:43.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:43.325 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:39:43.376 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:39:43.377 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:39:43.446 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:39:43.447 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:39:43.447 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:39:43.447 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:39:43.447 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:39:43.448 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:39:43.449 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:43.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:43.449 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:39:43.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:43.449 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:39:43.450 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:43.450 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:43.450 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.485421_716.1", 2022-01-11T14:39:43.450 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:43.451 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:39:43.451 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:43.451 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.485421Z", 2022-01-11T14:39:43.451 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:43.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:43.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:43.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:43.452 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:39:43.452 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:43.453 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:43.453 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:39:43.453 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:43.453 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:43.454 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:43.454 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:39:43.454 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:43.454 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:43.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:43.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:43.455 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:39:43.455 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:43.455 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:43.456 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:43.456 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:43.456 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:43.456 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:43.457 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:43.457 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:43.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:43.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:43.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:43.458 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:39:43.458 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:43.459 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:43.459 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:43.459 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:43.459 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:43.459 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:43.460 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:43.460 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:43.460 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:43.460 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:43.461 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:39:43.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:43.461 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:39:43.461 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:39:43.461 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:39:43.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:43.462 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:39:43.463 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.485421_716.1 2022-01-11T14:39:48.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:48.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:48.466 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:39:48.564 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:39:48.565 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:39:48.794 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:39:48.795 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:39:48.795 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:39:48.795 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:39:48.795 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:39:48.796 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:39:48.796 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:48.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:48.796 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:39:48.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:48.797 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:39:48.797 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:48.798 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:48.798 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:39:48.798 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:48.798 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:39:48.798 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:48.799 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:39:48.799 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:48.799 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:48.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:48.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:48.800 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:39:48.800 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:48.800 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:48.801 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:39:48.801 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:48.801 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:48.802 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:48.802 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:39:48.802 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:48.802 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:48.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:48.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:48.803 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:39:48.803 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:48.803 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:48.804 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:48.804 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:48.804 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:48.804 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:48.805 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:48.805 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:48.805 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:48.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:48.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:48.806 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:39:48.806 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:48.806 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:48.806 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:48.807 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:48.807 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:48.807 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:48.807 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:48.808 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:48.808 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:48.808 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:48.808 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:39:48.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:48.809 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:39:48.809 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:39:48.809 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:39:48.809 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:48.810 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:39:48.811 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:39:53.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:53.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:53.814 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:39:53.864 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:39:53.864 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:39:54.107 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:39:54.107 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:39:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:39:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:39:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:39:54.108 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:39:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:39:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:54.109 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:39:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.476921_714.1", 2022-01-11T14:39:54.110 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:39:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.476921Z", 2022-01-11T14:39:54.111 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:54.112 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:39:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:39:54.113 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:39:54.114 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:39:54.115 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:54.116 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:54.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:39:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:54.118 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:54.119 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:54.120 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:39:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:39:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:39:54.121 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:39:54.122 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:54.122 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:39:54.122 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.476921_714.1 2022-01-11T14:39:59.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:59.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:39:59.125 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:39:59.174 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:39:59.175 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:39:59.241 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:39:59.242 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:39:59.242 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:39:59.242 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:39:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:39:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:39:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:39:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:39:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:39:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:59.246 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:39:59.246 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:59.246 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:39:59.246 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:59.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:59.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:59.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:59.247 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:39:59.247 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:39:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:39:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:39:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:39:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:39:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:39:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:39:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:39:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:39:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:39:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:39:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:39:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:39:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:39:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:39:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:39:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:39:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:39:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:39:59.257 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:39:59.258 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:40:04.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:40:04.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:40:04.261 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:40:04.305 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:40:04.305 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:40:04.382 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:40:04.382 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:40:04.383 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:40:04.383 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:40:04.383 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:40:04.383 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:40:04.383 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:04.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:04.384 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:40:04.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:40:04.384 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:40:04.385 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:40:04.385 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:40:04.385 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910391.877188_736.1", 2022-01-11T14:40:04.385 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:40:04.386 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:40:04.386 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:40:04.386 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:11.877188Z", 2022-01-11T14:40:04.386 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:04.387 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:04.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:04.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:40:04.387 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:40:04.388 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:40:04.388 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:40:04.388 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:40:04.388 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:40:04.388 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:40:04.389 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:40:04.389 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:40:04.389 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:04.389 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:04.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:04.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:40:04.390 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:40:04.390 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:40:04.391 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:40:04.391 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:40:04.391 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:40:04.391 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:40:04.391 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:40:04.392 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:40:04.392 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:04.392 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:04.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:04.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:40:04.393 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:40:04.393 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:40:04.393 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:40:04.393 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:40:04.394 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:40:04.394 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:40:04.394 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:40:04.394 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:40:04.394 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:04.395 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:04.395 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:04.395 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:40:04.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:04.396 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:40:04.396 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:40:04.396 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:40:04.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:04.396 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:40:04.398 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910391.877188_736.1 2022-01-11T14:40:09.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:40:09.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:40:09.400 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:40:09.448 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:40:09.449 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:40:09.538 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:40:09.538 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:40:09.539 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:40:09.539 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:40:09.539 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:40:09.540 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:40:09.540 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:09.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:09.543 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:40:09.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:40:09.544 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:40:09.545 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:40:09.545 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:40:09.545 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.404090_712.1", 2022-01-11T14:40:09.545 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:40:09.546 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:40:09.546 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:40:09.546 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.404090Z", 2022-01-11T14:40:09.546 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:09.547 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:09.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:09.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:40:09.547 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:40:09.548 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:40:09.548 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:40:09.548 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:40:09.548 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:40:09.548 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:40:09.549 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:40:09.549 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:40:09.549 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:09.549 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:09.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:09.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:40:09.550 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:40:09.550 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:40:09.551 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:40:09.551 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:40:09.551 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:40:09.551 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:40:09.552 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:40:09.552 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:40:09.552 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:09.553 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:09.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:09.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:40:09.553 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:40:09.554 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:40:09.554 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:40:09.554 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:40:09.554 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:40:09.555 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:40:09.555 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:40:09.555 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:40:09.555 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:09.556 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:09.556 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:09.556 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:40:09.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:09.557 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:40:09.557 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:40:09.557 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:40:09.557 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:09.558 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:40:09.559 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.404090_712.1 2022-01-11T14:40:14.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:40:14.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T14:40:14.562 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T14:40:14.659 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:40:14.660 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:40:14.961 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2022-01-11T14:40:14.962 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2022-01-11T14:40:14.962 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2022-01-11T14:40:14.962 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2022-01-11T14:40:14.962 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2022-01-11T14:40:14.963 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:40:14.963 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:14.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:14.963 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2022-01-11T14:40:14.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:40:14.964 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2022-01-11T14:40:14.964 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:40:14.964 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:40:14.964 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910389.258675_705.1", 2022-01-11T14:40:14.965 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:40:14.965 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2022-01-11T14:40:14.965 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:40:14.965 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:13:09.258675Z", 2022-01-11T14:40:14.966 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:14.966 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:14.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:14.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:40:14.967 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2022-01-11T14:40:14.967 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:40:14.967 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:40:14.967 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1641910016.595927_81.1", 2022-01-11T14:40:14.967 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:40:14.968 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:40:14.968 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:40:14.968 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2022-01-11T14:06:56.595927Z", 2022-01-11T14:40:14.968 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:14.969 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:14.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:14.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:40:14.969 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2022-01-11T14:40:14.969 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:40:14.970 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:40:14.970 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:40:14.970 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:40:14.970 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:40:14.971 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:40:14.971 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:40:14.971 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:14.971 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:14.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:14.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2022-01-11T14:40:14.972 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2022-01-11T14:40:14.972 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2022-01-11T14:40:14.972 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2022-01-11T14:40:14.973 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2022-01-11T14:40:14.973 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2022-01-11T14:40:14.973 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2022-01-11T14:40:14.973 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2022-01-11T14:40:14.973 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2022-01-11T14:40:14.974 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2022-01-11T14:40:14.974 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:14.974 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:14.974 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2022-01-11T14:40:14.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2022-01-11T14:40:14.975 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2022-01-11T14:40:14.975 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2022-01-11T14:40:14.975 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2022-01-11T14:40:14.975 INFO:teuthology.orchestra.run.smithi172.stdout: } 2022-01-11T14:40:14.976 INFO:teuthology.orchestra.run.smithi172.stdout:} 2022-01-11T14:40:14.977 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1641911394.664935_798.1 target=1_1641910389.258675_705.1 2022-01-11T14:40:19.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... FAIL 2022-01-11T14:40:19.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:40:19.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:40:19.981 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:40:20.026 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:40:20.026 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:40:20.052 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:40:20.052 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "09609221-c7ca-4071-87d2-c98ebbe8e06b", 2022-01-11T14:40:20.053 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "remove_zg", 2022-01-11T14:40:20.053 INFO:teuthology.orchestra.run.smithi161.stdout: "api_name": "remove_zg", 2022-01-11T14:40:20.053 INFO:teuthology.orchestra.run.smithi161.stdout: "is_master": "false", 2022-01-11T14:40:20.054 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [], 2022-01-11T14:40:20.054 INFO:teuthology.orchestra.run.smithi161.stdout: "hostnames": [], 2022-01-11T14:40:20.054 INFO:teuthology.orchestra.run.smithi161.stdout: "hostnames_s3website": [], 2022-01-11T14:40:20.054 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zone": "", 2022-01-11T14:40:20.055 INFO:teuthology.orchestra.run.smithi161.stdout: "zones": [], 2022-01-11T14:40:20.055 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_targets": [], 2022-01-11T14:40:20.055 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "", 2022-01-11T14:40:20.055 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:40:20.056 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_policy": { 2022-01-11T14:40:20.056 INFO:teuthology.orchestra.run.smithi161.stdout: "groups": [] 2022-01-11T14:40:20.056 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.056 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:40:20.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:40:20.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:40:20.058 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:40:20.142 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:40:20.143 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:40:20.171 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:40:20.171 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "457e5920-b52f-42ba-a006-d15a05b05dac", 2022-01-11T14:40:20.171 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "remove", 2022-01-11T14:40:20.172 INFO:teuthology.orchestra.run.smithi161.stdout: "domain_root": "remove.rgw.meta:root", 2022-01-11T14:40:20.172 INFO:teuthology.orchestra.run.smithi161.stdout: "control_pool": "remove.rgw.control", 2022-01-11T14:40:20.172 INFO:teuthology.orchestra.run.smithi161.stdout: "gc_pool": "remove.rgw.log:gc", 2022-01-11T14:40:20.172 INFO:teuthology.orchestra.run.smithi161.stdout: "lc_pool": "remove.rgw.log:lc", 2022-01-11T14:40:20.173 INFO:teuthology.orchestra.run.smithi161.stdout: "log_pool": "remove.rgw.log", 2022-01-11T14:40:20.173 INFO:teuthology.orchestra.run.smithi161.stdout: "intent_log_pool": "remove.rgw.log:intent", 2022-01-11T14:40:20.173 INFO:teuthology.orchestra.run.smithi161.stdout: "usage_log_pool": "remove.rgw.log:usage", 2022-01-11T14:40:20.173 INFO:teuthology.orchestra.run.smithi161.stdout: "roles_pool": "remove.rgw.meta:roles", 2022-01-11T14:40:20.174 INFO:teuthology.orchestra.run.smithi161.stdout: "reshard_pool": "remove.rgw.log:reshard", 2022-01-11T14:40:20.174 INFO:teuthology.orchestra.run.smithi161.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2022-01-11T14:40:20.174 INFO:teuthology.orchestra.run.smithi161.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2022-01-11T14:40:20.174 INFO:teuthology.orchestra.run.smithi161.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2022-01-11T14:40:20.175 INFO:teuthology.orchestra.run.smithi161.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2022-01-11T14:40:20.175 INFO:teuthology.orchestra.run.smithi161.stdout: "otp_pool": "remove.rgw.otp", 2022-01-11T14:40:20.175 INFO:teuthology.orchestra.run.smithi161.stdout: "system_key": { 2022-01-11T14:40:20.175 INFO:teuthology.orchestra.run.smithi161.stdout: "access_key": "", 2022-01-11T14:40:20.175 INFO:teuthology.orchestra.run.smithi161.stdout: "secret_key": "" 2022-01-11T14:40:20.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.176 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_pools": [ 2022-01-11T14:40:20.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.176 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "default-placement", 2022-01-11T14:40:20.177 INFO:teuthology.orchestra.run.smithi161.stdout: "val": { 2022-01-11T14:40:20.177 INFO:teuthology.orchestra.run.smithi161.stdout: "index_pool": "remove.rgw.buckets.index", 2022-01-11T14:40:20.177 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": { 2022-01-11T14:40:20.177 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD": { 2022-01-11T14:40:20.178 INFO:teuthology.orchestra.run.smithi161.stdout: "data_pool": "remove.rgw.buckets.data" 2022-01-11T14:40:20.178 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.178 INFO:teuthology.orchestra.run.smithi161.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2022-01-11T14:40:20.179 INFO:teuthology.orchestra.run.smithi161.stdout: "index_type": 0 2022-01-11T14:40:20.179 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.179 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.179 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.180 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:40:20.180 INFO:teuthology.orchestra.run.smithi161.stdout: "notif_pool": "remove.rgw.log:notif" 2022-01-11T14:40:20.180 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:40:20.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:40:20.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:40:20.183 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:40:20.264 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:40:20.264 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:40:20.346 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:40:20.346 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:40:20.346 INFO:teuthology.orchestra.run.smithi161.stdout: "epoch": 6, 2022-01-11T14:40:20.347 INFO:teuthology.orchestra.run.smithi161.stdout: "predecessor_uuid": "bef33aea-c28e-4d3d-b156-c57e2ca93938", 2022-01-11T14:40:20.347 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": [], 2022-01-11T14:40:20.347 INFO:teuthology.orchestra.run.smithi161.stdout: "period_map": { 2022-01-11T14:40:20.347 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:40:20.348 INFO:teuthology.orchestra.run.smithi161.stdout: "zonegroups": [ 2022-01-11T14:40:20.348 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.348 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "09609221-c7ca-4071-87d2-c98ebbe8e06b", 2022-01-11T14:40:20.348 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "remove_zg", 2022-01-11T14:40:20.349 INFO:teuthology.orchestra.run.smithi161.stdout: "api_name": "remove_zg", 2022-01-11T14:40:20.349 INFO:teuthology.orchestra.run.smithi161.stdout: "is_master": "false", 2022-01-11T14:40:20.349 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [], 2022-01-11T14:40:20.349 INFO:teuthology.orchestra.run.smithi161.stdout: "hostnames": [], 2022-01-11T14:40:20.350 INFO:teuthology.orchestra.run.smithi161.stdout: "hostnames_s3website": [], 2022-01-11T14:40:20.350 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zone": "457e5920-b52f-42ba-a006-d15a05b05dac", 2022-01-11T14:40:20.350 INFO:teuthology.orchestra.run.smithi161.stdout: "zones": [ 2022-01-11T14:40:20.350 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.351 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "457e5920-b52f-42ba-a006-d15a05b05dac", 2022-01-11T14:40:20.351 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "remove", 2022-01-11T14:40:20.351 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [], 2022-01-11T14:40:20.351 INFO:teuthology.orchestra.run.smithi161.stdout: "log_meta": "false", 2022-01-11T14:40:20.352 INFO:teuthology.orchestra.run.smithi161.stdout: "log_data": "false", 2022-01-11T14:40:20.352 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:40:20.352 INFO:teuthology.orchestra.run.smithi161.stdout: "read_only": "false", 2022-01-11T14:40:20.353 INFO:teuthology.orchestra.run.smithi161.stdout: "tier_type": "", 2022-01-11T14:40:20.353 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from_all": "true", 2022-01-11T14:40:20.353 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from": [], 2022-01-11T14:40:20.353 INFO:teuthology.orchestra.run.smithi161.stdout: "redirect_zone": "" 2022-01-11T14:40:20.354 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.354 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.354 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_targets": [ 2022-01-11T14:40:20.354 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.355 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "default-placement", 2022-01-11T14:40:20.355 INFO:teuthology.orchestra.run.smithi161.stdout: "tags": [], 2022-01-11T14:40:20.355 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": [ 2022-01-11T14:40:20.355 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD" 2022-01-11T14:40:20.355 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:40:20.356 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.356 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.356 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "default-placement", 2022-01-11T14:40:20.356 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:40:20.357 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_policy": { 2022-01-11T14:40:20.357 INFO:teuthology.orchestra.run.smithi161.stdout: "groups": [] 2022-01-11T14:40:20.357 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.357 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.358 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.358 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "test-zonegroup", 2022-01-11T14:40:20.358 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zonegroup", 2022-01-11T14:40:20.359 INFO:teuthology.orchestra.run.smithi161.stdout: "api_name": "", 2022-01-11T14:40:20.359 INFO:teuthology.orchestra.run.smithi161.stdout: "is_master": "true", 2022-01-11T14:40:20.359 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:40:20.359 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:40:20.359 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.360 INFO:teuthology.orchestra.run.smithi161.stdout: "hostnames": [], 2022-01-11T14:40:20.360 INFO:teuthology.orchestra.run.smithi161.stdout: "hostnames_s3website": [], 2022-01-11T14:40:20.360 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zone": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:40:20.360 INFO:teuthology.orchestra.run.smithi161.stdout: "zones": [ 2022-01-11T14:40:20.361 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.361 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:40:20.361 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone2", 2022-01-11T14:40:20.362 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:40:20.362 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2022-01-11T14:40:20.362 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.363 INFO:teuthology.orchestra.run.smithi161.stdout: "log_meta": "false", 2022-01-11T14:40:20.363 INFO:teuthology.orchestra.run.smithi161.stdout: "log_data": "true", 2022-01-11T14:40:20.363 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:40:20.363 INFO:teuthology.orchestra.run.smithi161.stdout: "read_only": "false", 2022-01-11T14:40:20.363 INFO:teuthology.orchestra.run.smithi161.stdout: "tier_type": "", 2022-01-11T14:40:20.364 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from_all": "false", 2022-01-11T14:40:20.364 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from": [], 2022-01-11T14:40:20.364 INFO:teuthology.orchestra.run.smithi161.stdout: "redirect_zone": "" 2022-01-11T14:40:20.364 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.365 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.365 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:40:20.365 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone1", 2022-01-11T14:40:20.365 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:40:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:40:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: "log_meta": "false", 2022-01-11T14:40:20.366 INFO:teuthology.orchestra.run.smithi161.stdout: "log_data": "true", 2022-01-11T14:40:20.367 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:40:20.367 INFO:teuthology.orchestra.run.smithi161.stdout: "read_only": "false", 2022-01-11T14:40:20.367 INFO:teuthology.orchestra.run.smithi161.stdout: "tier_type": "", 2022-01-11T14:40:20.367 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from_all": "true", 2022-01-11T14:40:20.367 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from": [], 2022-01-11T14:40:20.368 INFO:teuthology.orchestra.run.smithi161.stdout: "redirect_zone": "" 2022-01-11T14:40:20.368 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.368 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.368 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "a285481c-4c5c-487e-978c-9cbad192af6e", 2022-01-11T14:40:20.369 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone3", 2022-01-11T14:40:20.369 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:40:20.369 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi161.front.sepia.ceph.com:8001" 2022-01-11T14:40:20.369 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.369 INFO:teuthology.orchestra.run.smithi161.stdout: "log_meta": "false", 2022-01-11T14:40:20.370 INFO:teuthology.orchestra.run.smithi161.stdout: "log_data": "true", 2022-01-11T14:40:20.370 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:40:20.370 INFO:teuthology.orchestra.run.smithi161.stdout: "read_only": "false", 2022-01-11T14:40:20.370 INFO:teuthology.orchestra.run.smithi161.stdout: "tier_type": "", 2022-01-11T14:40:20.372 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from_all": "true", 2022-01-11T14:40:20.372 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from": [], 2022-01-11T14:40:20.372 INFO:teuthology.orchestra.run.smithi161.stdout: "redirect_zone": "" 2022-01-11T14:40:20.373 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.373 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.373 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "ccb5abc6-5490-49c7-b386-02859e29748c", 2022-01-11T14:40:20.374 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone4", 2022-01-11T14:40:20.374 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:40:20.374 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2022-01-11T14:40:20.374 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.374 INFO:teuthology.orchestra.run.smithi161.stdout: "log_meta": "false", 2022-01-11T14:40:20.375 INFO:teuthology.orchestra.run.smithi161.stdout: "log_data": "true", 2022-01-11T14:40:20.375 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:40:20.375 INFO:teuthology.orchestra.run.smithi161.stdout: "read_only": "false", 2022-01-11T14:40:20.375 INFO:teuthology.orchestra.run.smithi161.stdout: "tier_type": "pubsub", 2022-01-11T14:40:20.376 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from_all": "false", 2022-01-11T14:40:20.376 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from": [ 2022-01-11T14:40:20.376 INFO:teuthology.orchestra.run.smithi161.stdout: "test-zone1" 2022-01-11T14:40:20.376 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.377 INFO:teuthology.orchestra.run.smithi161.stdout: "redirect_zone": "" 2022-01-11T14:40:20.377 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.377 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.377 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_targets": [ 2022-01-11T14:40:20.377 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.378 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "default-placement", 2022-01-11T14:40:20.378 INFO:teuthology.orchestra.run.smithi161.stdout: "tags": [], 2022-01-11T14:40:20.378 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": [ 2022-01-11T14:40:20.379 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD" 2022-01-11T14:40:20.379 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:40:20.379 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.379 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.379 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "default-placement", 2022-01-11T14:40:20.380 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:40:20.380 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_policy": { 2022-01-11T14:40:20.380 INFO:teuthology.orchestra.run.smithi161.stdout: "groups": [] 2022-01-11T14:40:20.380 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.381 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.381 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.381 INFO:teuthology.orchestra.run.smithi161.stdout: "short_zone_ids": [ 2022-01-11T14:40:20.381 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.382 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:40:20.382 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 2774529435 2022-01-11T14:40:20.382 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.382 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.382 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "457e5920-b52f-42ba-a006-d15a05b05dac", 2022-01-11T14:40:20.383 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 2809237460 2022-01-11T14:40:20.383 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.383 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.384 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:40:20.384 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 618671346 2022-01-11T14:40:20.384 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.384 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.385 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "a285481c-4c5c-487e-978c-9cbad192af6e", 2022-01-11T14:40:20.385 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 544948410 2022-01-11T14:40:20.385 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.385 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.386 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "ccb5abc6-5490-49c7-b386-02859e29748c", 2022-01-11T14:40:20.386 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 2309622666 2022-01-11T14:40:20.386 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.386 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:40:20.386 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.387 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zonegroup": "test-zonegroup", 2022-01-11T14:40:20.387 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zone": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:40:20.387 INFO:teuthology.orchestra.run.smithi161.stdout: "period_config": { 2022-01-11T14:40:20.387 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_quota": { 2022-01-11T14:40:20.388 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2022-01-11T14:40:20.388 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2022-01-11T14:40:20.388 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2022-01-11T14:40:20.388 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2022-01-11T14:40:20.389 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2022-01-11T14:40:20.389 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.389 INFO:teuthology.orchestra.run.smithi161.stdout: "user_quota": { 2022-01-11T14:40:20.389 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2022-01-11T14:40:20.389 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2022-01-11T14:40:20.390 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2022-01-11T14:40:20.390 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2022-01-11T14:40:20.390 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2022-01-11T14:40:20.390 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.391 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.391 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:40:20.391 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_name": "test-realm", 2022-01-11T14:40:20.392 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:40:20.392 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:40:20.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:40:20.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:40:20.394 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:40:20.452 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:40:20.452 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:40:20.482 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:40:20.484+0000 7f6b17e67500 0 zone id 457e5920-b52f-42ba-a006-d15a05b05dac is not a part of zonegroup test-zonegroup 2022-01-11T14:40:20.483 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:40:20.485+0000 7f6b17e67500 0 zone id 457e5920-b52f-42ba-a006-d15a05b05dac is not a part of zonegroup default 2022-01-11T14:40:20.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:40:20.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:40:20.494 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:40:20.579 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:40:20.580 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:40:20.616 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T14:40:20.618+0000 7f6b5e47f500 0 ERROR: zonegroup remove_zg has a non existent master zone 2022-01-11T14:40:20.621 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-01-11T14:40:20.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:40:20.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:40:20.623 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:40:20.674 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:40:20.675 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:40:20.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:40:20.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2022-01-11T14:40:20.722 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2022-01-11T14:40:20.774 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T14:40:20.775 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T14:40:20.941 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-01-11T14:40:20.941 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:40:20.942 INFO:teuthology.orchestra.run.smithi161.stdout: "epoch": 7, 2022-01-11T14:40:20.942 INFO:teuthology.orchestra.run.smithi161.stdout: "predecessor_uuid": "bef33aea-c28e-4d3d-b156-c57e2ca93938", 2022-01-11T14:40:20.942 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_status": [], 2022-01-11T14:40:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: "period_map": { 2022-01-11T14:40:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "86901227-b7e5-4fef-8f4e-54cce97cf2a2", 2022-01-11T14:40:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: "zonegroups": [ 2022-01-11T14:40:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "test-zonegroup", 2022-01-11T14:40:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zonegroup", 2022-01-11T14:40:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: "api_name": "", 2022-01-11T14:40:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: "is_master": "true", 2022-01-11T14:40:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:40:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:40:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: "hostnames": [], 2022-01-11T14:40:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: "hostnames_s3website": [], 2022-01-11T14:40:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zone": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:40:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: "zones": [ 2022-01-11T14:40:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:40:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone2", 2022-01-11T14:40:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:40:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2022-01-11T14:40:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: "log_meta": "false", 2022-01-11T14:40:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: "log_data": "true", 2022-01-11T14:40:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:40:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: "read_only": "false", 2022-01-11T14:40:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: "tier_type": "", 2022-01-11T14:40:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from_all": "false", 2022-01-11T14:40:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from": [], 2022-01-11T14:40:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: "redirect_zone": "" 2022-01-11T14:40:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:40:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone1", 2022-01-11T14:40:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:40:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi161.front.sepia.ceph.com:8000" 2022-01-11T14:40:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: "log_meta": "false", 2022-01-11T14:40:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: "log_data": "true", 2022-01-11T14:40:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:40:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: "read_only": "false", 2022-01-11T14:40:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: "tier_type": "", 2022-01-11T14:40:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from_all": "true", 2022-01-11T14:40:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from": [], 2022-01-11T14:40:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: "redirect_zone": "" 2022-01-11T14:40:20.953 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.953 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "a285481c-4c5c-487e-978c-9cbad192af6e", 2022-01-11T14:40:20.953 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone3", 2022-01-11T14:40:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:40:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi161.front.sepia.ceph.com:8001" 2022-01-11T14:40:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: "log_meta": "false", 2022-01-11T14:40:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: "log_data": "true", 2022-01-11T14:40:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:40:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: "read_only": "false", 2022-01-11T14:40:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: "tier_type": "", 2022-01-11T14:40:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from_all": "true", 2022-01-11T14:40:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from": [], 2022-01-11T14:40:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: "redirect_zone": "" 2022-01-11T14:40:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: "id": "ccb5abc6-5490-49c7-b386-02859e29748c", 2022-01-11T14:40:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "test-zone4", 2022-01-11T14:40:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: "endpoints": [ 2022-01-11T14:40:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2022-01-11T14:40:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: "log_meta": "false", 2022-01-11T14:40:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: "log_data": "true", 2022-01-11T14:40:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_index_max_shards": 11, 2022-01-11T14:40:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: "read_only": "false", 2022-01-11T14:40:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: "tier_type": "pubsub", 2022-01-11T14:40:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from_all": "false", 2022-01-11T14:40:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_from": [ 2022-01-11T14:40:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: "test-zone1" 2022-01-11T14:40:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: "redirect_zone": "" 2022-01-11T14:40:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: "placement_targets": [ 2022-01-11T14:40:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: "name": "default-placement", 2022-01-11T14:40:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: "tags": [], 2022-01-11T14:40:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: "storage_classes": [ 2022-01-11T14:40:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: "STANDARD" 2022-01-11T14:40:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:40:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: "default_placement": "default-placement", 2022-01-11T14:40:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:40:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: "sync_policy": { 2022-01-11T14:40:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: "groups": [] 2022-01-11T14:40:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2022-01-11T14:40:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: "short_zone_ids": [ 2022-01-11T14:40:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:21.000 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "3b4f5cd7-b080-4871-ae14-6286289db7bf", 2022-01-11T14:40:21.000 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 2774529435 2022-01-11T14:40:21.000 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:21.001 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:21.001 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:40:21.001 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 618671346 2022-01-11T14:40:21.001 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:21.002 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:21.002 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "a285481c-4c5c-487e-978c-9cbad192af6e", 2022-01-11T14:40:21.002 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 544948410 2022-01-11T14:40:21.002 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: { 2022-01-11T14:40:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "ccb5abc6-5490-49c7-b386-02859e29748c", 2022-01-11T14:40:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 2309622666 2022-01-11T14:40:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:21.004 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-01-11T14:40:21.004 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:21.004 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zonegroup": "test-zonegroup", 2022-01-11T14:40:21.004 INFO:teuthology.orchestra.run.smithi161.stdout: "master_zone": "776aa573-af61-46ca-a8eb-de18a933b620", 2022-01-11T14:40:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: "period_config": { 2022-01-11T14:40:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: "bucket_quota": { 2022-01-11T14:40:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2022-01-11T14:40:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2022-01-11T14:40:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2022-01-11T14:40:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2022-01-11T14:40:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2022-01-11T14:40:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: "user_quota": { 2022-01-11T14:40:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: "enabled": false, 2022-01-11T14:40:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: "check_on_raw": false, 2022-01-11T14:40:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size": -1, 2022-01-11T14:40:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: "max_size_kb": 0, 2022-01-11T14:40:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: "max_objects": -1 2022-01-11T14:40:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: } 2022-01-11T14:40:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2022-01-11T14:40:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_id": "f6bbcad3-0f7d-4a5a-a1e4-1907fdff0669", 2022-01-11T14:40:21.009 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_name": "test-realm", 2022-01-11T14:40:21.009 INFO:teuthology.orchestra.run.smithi161.stdout: "realm_epoch": 2 2022-01-11T14:40:21.009 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-01-11T14:40:21.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2022-01-11T14:40:21.012 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-46 2022-01-11T14:40:21.740 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-47 2022-01-11T14:40:22.120 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=epmlcl-48 2022-01-11T14:40:22.122 INFO:tasks.rgw.c2.client.0.smithi172.stdout:*** Caught signal (Segmentation fault) ** 2022-01-11T14:40:22.122 INFO:tasks.rgw.c2.client.0.smithi172.stdout: in thread 7f97c5220700 thread_name:radosgw 2022-01-11T14:40:22.178 INFO:tasks.rgw.c1.client.1.smithi161.stdout:*** Caught signal (Segmentation fault) ** 2022-01-11T14:40:22.178 INFO:tasks.rgw.c1.client.1.smithi161.stdout: in thread 7fc1853fa700 thread_name:radosgw 2022-01-11T14:40:22.179 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-49 2022-01-11T14:40:22.213 INFO:tasks.rgw.c2.client.0.smithi172.stdout: ceph version 17.0.0-10004-g17f14252 (17f142526478a11e8f62d3933be552d546c4fd19) quincy (dev) 2022-01-11T14:40:22.213 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f98e0ac4b20] 2022-01-11T14:40:22.213 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 2: (OpsLogRados::log(req_state*, rgw_log_entry&)+0x123) [0x7f98e36206c3] 2022-01-11T14:40:22.213 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 3: (OpsLogManifold::log(req_state*, rgw_log_entry&)+0x3e) [0x7f98e361d77e] 2022-01-11T14:40:22.214 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 4: (rgw_log_op(RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSink*)+0xac2) [0x7f98e36213c2] 2022-01-11T14:40:22.214 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 5: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1861) [0x7f98e363f961] 2022-01-11T14:40:22.214 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 6: /lib64/libradosgw.so.2(+0x4c58ed) [0x7f98e35ad8ed] 2022-01-11T14:40:22.214 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 7: /lib64/libradosgw.so.2(+0x4c6f71) [0x7f98e35aef71] 2022-01-11T14:40:22.215 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 8: /lib64/libradosgw.so.2(+0x4c70ec) [0x7f98e35af0ec] 2022-01-11T14:40:22.215 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 9: make_fcontext() 2022-01-11T14:40:22.215 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2022-01-11T14:40:22.123+0000 7f97c5220700 -1 *** Caught signal (Segmentation fault) ** 2022-01-11T14:40:22.215 INFO:tasks.rgw.c2.client.0.smithi172.stdout: in thread 7f97c5220700 thread_name:radosgw 2022-01-11T14:40:22.216 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 2022-01-11T14:40:22.216 INFO:tasks.rgw.c2.client.0.smithi172.stdout: ceph version 17.0.0-10004-g17f14252 (17f142526478a11e8f62d3933be552d546c4fd19) quincy (dev) 2022-01-11T14:40:22.216 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f98e0ac4b20] 2022-01-11T14:40:22.216 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 2: (OpsLogRados::log(req_state*, rgw_log_entry&)+0x123) [0x7f98e36206c3] 2022-01-11T14:40:22.217 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 3: (OpsLogManifold::log(req_state*, rgw_log_entry&)+0x3e) [0x7f98e361d77e] 2022-01-11T14:40:22.217 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 4: (rgw_log_op(RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSink*)+0xac2) [0x7f98e36213c2] 2022-01-11T14:40:22.217 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 5: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1861) [0x7f98e363f961] 2022-01-11T14:40:22.217 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 6: /lib64/libradosgw.so.2(+0x4c58ed) [0x7f98e35ad8ed] 2022-01-11T14:40:22.218 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 7: /lib64/libradosgw.so.2(+0x4c6f71) [0x7f98e35aef71] 2022-01-11T14:40:22.218 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 8: /lib64/libradosgw.so.2(+0x4c70ec) [0x7f98e35af0ec] 2022-01-11T14:40:22.219 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 9: make_fcontext() 2022-01-11T14:40:22.219 INFO:tasks.rgw.c2.client.0.smithi172.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-11T14:40:22.219 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 2022-01-11T14:40:22.219 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 0> 2022-01-11T14:40:22.123+0000 7f97c5220700 -1 *** Caught signal (Segmentation fault) ** 2022-01-11T14:40:22.220 INFO:tasks.rgw.c2.client.0.smithi172.stdout: in thread 7f97c5220700 thread_name:radosgw 2022-01-11T14:40:22.220 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 2022-01-11T14:40:22.220 INFO:tasks.rgw.c2.client.0.smithi172.stdout: ceph version 17.0.0-10004-g17f14252 (17f142526478a11e8f62d3933be552d546c4fd19) quincy (dev) 2022-01-11T14:40:22.220 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f98e0ac4b20] 2022-01-11T14:40:22.220 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 2: (OpsLogRados::log(req_state*, rgw_log_entry&)+0x123) [0x7f98e36206c3] 2022-01-11T14:40:22.221 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 3: (OpsLogManifold::log(req_state*, rgw_log_entry&)+0x3e) [0x7f98e361d77e] 2022-01-11T14:40:22.221 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 4: (rgw_log_op(RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSink*)+0xac2) [0x7f98e36213c2] 2022-01-11T14:40:22.221 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 5: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1861) [0x7f98e363f961] 2022-01-11T14:40:22.221 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 6: /lib64/libradosgw.so.2(+0x4c58ed) [0x7f98e35ad8ed] 2022-01-11T14:40:22.222 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 7: /lib64/libradosgw.so.2(+0x4c6f71) [0x7f98e35aef71] 2022-01-11T14:40:22.222 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 8: /lib64/libradosgw.so.2(+0x4c70ec) [0x7f98e35af0ec] 2022-01-11T14:40:22.222 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 9: make_fcontext() 2022-01-11T14:40:22.222 INFO:tasks.rgw.c2.client.0.smithi172.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-11T14:40:22.223 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 2022-01-11T14:40:22.223 INFO:tasks.rgw.c2.client.0.smithi172.stdout: -90> 2022-01-11T14:40:22.123+0000 7f97c5220700 -1 *** Caught signal (Segmentation fault) ** 2022-01-11T14:40:22.223 INFO:tasks.rgw.c2.client.0.smithi172.stdout: in thread 7f97c5220700 thread_name:radosgw 2022-01-11T14:40:22.223 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 2022-01-11T14:40:22.223 INFO:tasks.rgw.c2.client.0.smithi172.stdout: ceph version 17.0.0-10004-g17f14252 (17f142526478a11e8f62d3933be552d546c4fd19) quincy (dev) 2022-01-11T14:40:22.224 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f98e0ac4b20] 2022-01-11T14:40:22.224 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 2: (OpsLogRados::log(req_state*, rgw_log_entry&)+0x123) [0x7f98e36206c3] 2022-01-11T14:40:22.224 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 3: (OpsLogManifold::log(req_state*, rgw_log_entry&)+0x3e) [0x7f98e361d77e] 2022-01-11T14:40:22.224 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 4: (rgw_log_op(RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSink*)+0xac2) [0x7f98e36213c2] 2022-01-11T14:40:22.225 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 5: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1861) [0x7f98e363f961] 2022-01-11T14:40:22.225 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 6: /lib64/libradosgw.so.2(+0x4c58ed) [0x7f98e35ad8ed] 2022-01-11T14:40:22.225 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 7: /lib64/libradosgw.so.2(+0x4c6f71) [0x7f98e35aef71] 2022-01-11T14:40:22.225 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 8: /lib64/libradosgw.so.2(+0x4c70ec) [0x7f98e35af0ec] 2022-01-11T14:40:22.226 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 9: make_fcontext() 2022-01-11T14:40:22.226 INFO:tasks.rgw.c2.client.0.smithi172.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-11T14:40:22.226 INFO:tasks.rgw.c2.client.0.smithi172.stdout: 2022-01-11T14:40:22.227 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=epmlcl-50 2022-01-11T14:40:22.228 INFO:tasks.rgw.c2.client.0.smithi172.stderr:daemon-helper: command crashed with signal 11 2022-01-11T14:40:22.232 INFO:tasks.rgw.c1.client.1.smithi161.stdout: ceph version 17.0.0-10004-g17f14252 (17f142526478a11e8f62d3933be552d546c4fd19) quincy (dev) 2022-01-11T14:40:22.232 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc21037db20] 2022-01-11T14:40:22.233 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 2: (OpsLogRados::log(req_state*, rgw_log_entry&)+0x123) [0x7fc212ed96c3] 2022-01-11T14:40:22.233 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 3: (OpsLogManifold::log(req_state*, rgw_log_entry&)+0x3e) [0x7fc212ed677e] 2022-01-11T14:40:22.233 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 4: (rgw_log_op(RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSink*)+0xac2) [0x7fc212eda3c2] 2022-01-11T14:40:22.233 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 5: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1861) [0x7fc212ef8961] 2022-01-11T14:40:22.233 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 6: /lib64/libradosgw.so.2(+0x4c58ed) [0x7fc212e668ed] 2022-01-11T14:40:22.234 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 7: /lib64/libradosgw.so.2(+0x4c6f71) [0x7fc212e67f71] 2022-01-11T14:40:22.234 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 8: /lib64/libradosgw.so.2(+0x4c70ec) [0x7fc212e680ec] 2022-01-11T14:40:22.234 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 9: make_fcontext() 2022-01-11T14:40:22.234 INFO:tasks.rgw.c1.client.1.smithi161.stdout:2022-01-11T14:40:22.181+0000 7fc1853fa700 -1 *** Caught signal (Segmentation fault) ** 2022-01-11T14:40:22.235 INFO:tasks.rgw.c1.client.1.smithi161.stdout: in thread 7fc1853fa700 thread_name:radosgw 2022-01-11T14:40:22.235 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 2022-01-11T14:40:22.235 INFO:tasks.rgw.c1.client.1.smithi161.stdout: ceph version 17.0.0-10004-g17f14252 (17f142526478a11e8f62d3933be552d546c4fd19) quincy (dev) 2022-01-11T14:40:22.235 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc21037db20] 2022-01-11T14:40:22.236 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 2: (OpsLogRados::log(req_state*, rgw_log_entry&)+0x123) [0x7fc212ed96c3] 2022-01-11T14:40:22.236 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 3: (OpsLogManifold::log(req_state*, rgw_log_entry&)+0x3e) [0x7fc212ed677e] 2022-01-11T14:40:22.236 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 4: (rgw_log_op(RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSink*)+0xac2) [0x7fc212eda3c2] 2022-01-11T14:40:22.236 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 5: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1861) [0x7fc212ef8961] 2022-01-11T14:40:22.237 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 6: /lib64/libradosgw.so.2(+0x4c58ed) [0x7fc212e668ed] 2022-01-11T14:40:22.237 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 7: /lib64/libradosgw.so.2(+0x4c6f71) [0x7fc212e67f71] 2022-01-11T14:40:22.237 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 8: /lib64/libradosgw.so.2(+0x4c70ec) [0x7fc212e680ec] 2022-01-11T14:40:22.237 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 9: make_fcontext() 2022-01-11T14:40:22.237 INFO:tasks.rgw.c1.client.1.smithi161.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-11T14:40:22.238 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 2022-01-11T14:40:22.238 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 0> 2022-01-11T14:40:22.181+0000 7fc1853fa700 -1 *** Caught signal (Segmentation fault) ** 2022-01-11T14:40:22.238 INFO:tasks.rgw.c1.client.1.smithi161.stdout: in thread 7fc1853fa700 thread_name:radosgw 2022-01-11T14:40:22.238 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 2022-01-11T14:40:22.239 INFO:tasks.rgw.c1.client.1.smithi161.stdout: ceph version 17.0.0-10004-g17f14252 (17f142526478a11e8f62d3933be552d546c4fd19) quincy (dev) 2022-01-11T14:40:22.239 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc21037db20] 2022-01-11T14:40:22.239 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 2: (OpsLogRados::log(req_state*, rgw_log_entry&)+0x123) [0x7fc212ed96c3] 2022-01-11T14:40:22.239 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 3: (OpsLogManifold::log(req_state*, rgw_log_entry&)+0x3e) [0x7fc212ed677e] 2022-01-11T14:40:22.240 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 4: (rgw_log_op(RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSink*)+0xac2) [0x7fc212eda3c2] 2022-01-11T14:40:22.240 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 5: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1861) [0x7fc212ef8961] 2022-01-11T14:40:22.240 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 6: /lib64/libradosgw.so.2(+0x4c58ed) [0x7fc212e668ed] 2022-01-11T14:40:22.240 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 7: /lib64/libradosgw.so.2(+0x4c6f71) [0x7fc212e67f71] 2022-01-11T14:40:22.240 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 8: /lib64/libradosgw.so.2(+0x4c70ec) [0x7fc212e680ec] 2022-01-11T14:40:22.241 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 9: make_fcontext() 2022-01-11T14:40:22.241 INFO:tasks.rgw.c1.client.1.smithi161.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-11T14:40:22.241 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 2022-01-11T14:40:22.241 INFO:tasks.rgw.c1.client.1.smithi161.stdout: -4> 2022-01-11T14:40:22.181+0000 7fc1853fa700 -1 *** Caught signal (Segmentation fault) ** 2022-01-11T14:40:22.241 INFO:tasks.rgw.c1.client.1.smithi161.stdout: in thread 7fc1853fa700 thread_name:radosgw 2022-01-11T14:40:22.242 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 2022-01-11T14:40:22.242 INFO:tasks.rgw.c1.client.1.smithi161.stdout: ceph version 17.0.0-10004-g17f14252 (17f142526478a11e8f62d3933be552d546c4fd19) quincy (dev) 2022-01-11T14:40:22.242 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc21037db20] 2022-01-11T14:40:22.242 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 2: (OpsLogRados::log(req_state*, rgw_log_entry&)+0x123) [0x7fc212ed96c3] 2022-01-11T14:40:22.243 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 3: (OpsLogManifold::log(req_state*, rgw_log_entry&)+0x3e) [0x7fc212ed677e] 2022-01-11T14:40:22.243 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 4: (rgw_log_op(RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSink*)+0xac2) [0x7fc212eda3c2] 2022-01-11T14:40:22.243 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 5: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1861) [0x7fc212ef8961] 2022-01-11T14:40:22.243 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 6: /lib64/libradosgw.so.2(+0x4c58ed) [0x7fc212e668ed] 2022-01-11T14:40:22.244 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 7: /lib64/libradosgw.so.2(+0x4c6f71) [0x7fc212e67f71] 2022-01-11T14:40:22.244 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 8: /lib64/libradosgw.so.2(+0x4c70ec) [0x7fc212e680ec] 2022-01-11T14:40:22.244 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 9: make_fcontext() 2022-01-11T14:40:22.244 INFO:tasks.rgw.c1.client.1.smithi161.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-01-11T14:40:22.244 INFO:tasks.rgw.c1.client.1.smithi161.stdout: 2022-01-11T14:40:22.245 INFO:tasks.rgw.c1.client.1.smithi161.stderr:daemon-helper: command crashed with signal 11 2022-01-11T14:40:25.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2022-01-11T14:40:26.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2022-01-11T14:40:31.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~6s 2022-01-11T14:40:31.684 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~6s 2022-01-11T14:40:36.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~11s 2022-01-11T14:40:37.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~11s 2022-01-11T14:40:42.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~17s 2022-01-11T14:40:42.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~17s 2022-01-11T14:40:48.122 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~22s 2022-01-11T14:40:48.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~22s 2022-01-11T14:40:53.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~28s 2022-01-11T14:40:54.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~28s 2022-01-11T14:40:59.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~34s 2022-01-11T14:40:59.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~34s 2022-01-11T14:41:04.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~39s 2022-01-11T14:41:05.318 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~39s 2022-01-11T14:41:10.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~45s 2022-01-11T14:41:10.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~45s 2022-01-11T14:41:16.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~50s 2022-01-11T14:41:16.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~50s 2022-01-11T14:41:21.762 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~56s 2022-01-11T14:41:22.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~56s 2022-01-11T14:41:27.368 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~62s 2022-01-11T14:41:27.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~62s 2022-01-11T14:41:32.974 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~67s 2022-01-11T14:41:33.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~67s 2022-01-11T14:41:38.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~73s 2022-01-11T14:41:38.958 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~73s 2022-01-11T14:41:44.187 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~78s 2022-01-11T14:41:44.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~78s 2022-01-11T14:41:49.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~84s 2022-01-11T14:41:50.170 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~84s 2022-01-11T14:41:55.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~90s 2022-01-11T14:41:55.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~90s 2022-01-11T14:42:01.007 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~95s 2022-01-11T14:42:01.382 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~95s 2022-01-11T14:42:06.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~101s 2022-01-11T14:42:06.989 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~101s 2022-01-11T14:42:12.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~107s 2022-01-11T14:42:12.595 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~107s 2022-01-11T14:42:17.827 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~112s 2022-01-11T14:42:18.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~112s 2022-01-11T14:42:23.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~118s 2022-01-11T14:42:23.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~118s 2022-01-11T14:42:29.040 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~123s 2022-01-11T14:42:29.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~123s 2022-01-11T14:42:34.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~129s 2022-01-11T14:42:35.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~129s 2022-01-11T14:42:40.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~135s 2022-01-11T14:42:40.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~135s 2022-01-11T14:42:45.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~140s 2022-01-11T14:42:46.230 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~140s 2022-01-11T14:42:51.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~146s 2022-01-11T14:42:51.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~146s 2022-01-11T14:42:57.070 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~151s 2022-01-11T14:42:57.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~151s 2022-01-11T14:43:02.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~157s 2022-01-11T14:43:03.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~157s 2022-01-11T14:43:08.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~163s 2022-01-11T14:43:08.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~163s 2022-01-11T14:43:13.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~168s 2022-01-11T14:43:14.259 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~168s 2022-01-11T14:43:19.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~174s 2022-01-11T14:43:19.864 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~174s 2022-01-11T14:43:25.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~179s 2022-01-11T14:43:25.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~179s 2022-01-11T14:43:30.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~185s 2022-01-11T14:43:31.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~185s 2022-01-11T14:43:36.315 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~191s 2022-01-11T14:43:36.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~191s 2022-01-11T14:43:41.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~196s 2022-01-11T14:43:42.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~196s 2022-01-11T14:43:47.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~202s 2022-01-11T14:43:47.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~202s 2022-01-11T14:43:53.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~207s 2022-01-11T14:43:53.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~207s 2022-01-11T14:43:58.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~213s 2022-01-11T14:43:59.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~213s 2022-01-11T14:44:04.345 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~219s 2022-01-11T14:44:04.709 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~219s 2022-01-11T14:44:09.951 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~224s 2022-01-11T14:44:10.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~224s 2022-01-11T14:44:15.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~230s 2022-01-11T14:44:15.920 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~230s 2022-01-11T14:44:21.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~235s 2022-01-11T14:44:21.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~235s 2022-01-11T14:44:26.769 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~241s 2022-01-11T14:44:27.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~241s 2022-01-11T14:44:32.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~247s 2022-01-11T14:44:32.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~247s 2022-01-11T14:44:37.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~252s 2022-01-11T14:44:38.342 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~252s 2022-01-11T14:44:43.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~258s 2022-01-11T14:44:43.946 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~258s 2022-01-11T14:44:49.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~263s 2022-01-11T14:44:49.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~263s 2022-01-11T14:44:54.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~269s 2022-01-11T14:44:55.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~269s 2022-01-11T14:45:00.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~275s 2022-01-11T14:45:00.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~275s 2022-01-11T14:45:06.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~280s 2022-01-11T14:45:06.367 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~280s 2022-01-11T14:45:11.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~286s 2022-01-11T14:45:11.972 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~286s 2022-01-11T14:45:17.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~292s 2022-01-11T14:45:17.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~292s 2022-01-11T14:45:22.830 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~297s 2022-01-11T14:45:23.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~297s 2022-01-11T14:45:28.436 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~303s 2022-01-11T14:45:28.436 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2022-01-11T14:45:28.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~303s 2022-01-11T14:45:28.788 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2022-01-11T14:45:29.041 INFO:tasks.ceph.c2.osd.0:Sent signal 15 2022-01-11T14:45:29.044 INFO:tasks.ceph.c2.osd.1:Sent signal 15 2022-01-11T14:45:29.044 INFO:tasks.ceph.c2.osd.2:Sent signal 15 2022-01-11T14:45:29.045 INFO:tasks.ceph.c2.mon.a:Sent signal 15 2022-01-11T14:45:29.045 INFO:tasks.rgw.c2.client.1:Sent signal 15 2022-01-11T14:45:29.046 INFO:tasks.ceph.c2.mgr.x:Sent signal 15 2022-01-11T14:45:29.047 INFO:tasks.ceph.c2.osd.0.smithi172.stderr:2022-01-11T14:45:29.044+0000 7fb17f3ce700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 0 (PID: 32117) UID: 0 2022-01-11T14:45:29.047 INFO:tasks.ceph.c2.osd.0.smithi172.stderr:2022-01-11T14:45:29.044+0000 7fb17f3ce700 -1 osd.0 57 *** Got signal Terminated *** 2022-01-11T14:45:29.047 INFO:tasks.ceph.c2.osd.0.smithi172.stderr:2022-01-11T14:45:29.044+0000 7fb17f3ce700 -1 osd.0 57 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-01-11T14:45:29.048 INFO:tasks.ceph.c2.osd.1.smithi172.stderr:2022-01-11T14:45:29.047+0000 7f50f8dc3700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 1 (PID: 32119) UID: 0 2022-01-11T14:45:29.048 INFO:tasks.ceph.c2.osd.1.smithi172.stderr:2022-01-11T14:45:29.047+0000 7f50f8dc3700 -1 osd.1 57 *** Got signal Terminated *** 2022-01-11T14:45:29.048 INFO:tasks.ceph.c2.osd.1.smithi172.stderr:2022-01-11T14:45:29.047+0000 7f50f8dc3700 -1 osd.1 57 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-01-11T14:45:29.049 INFO:tasks.ceph.c2.osd.2.smithi172.stderr:2022-01-11T14:45:29.047+0000 7f79039f1700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c2 -i 2 (PID: 32122) UID: 0 2022-01-11T14:45:29.049 INFO:tasks.ceph.c2.osd.2.smithi172.stderr:2022-01-11T14:45:29.047+0000 7f79039f1700 -1 osd.2 57 *** Got signal Terminated *** 2022-01-11T14:45:29.050 INFO:tasks.ceph.c2.osd.2.smithi172.stderr:2022-01-11T14:45:29.047+0000 7f79039f1700 -1 osd.2 57 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-01-11T14:45:29.050 INFO:tasks.ceph.c2.mon.a.smithi172.stderr:2022-01-11T14:45:29.048+0000 7f0f8886c700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c2 -i a (PID: 31783) UID: 0 2022-01-11T14:45:29.051 INFO:tasks.ceph.c2.mon.a.smithi172.stderr:2022-01-11T14:45:29.048+0000 7f0f8886c700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-01-11T14:45:29.051 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:45:29.048+0000 7f3da7c1a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 38296) UID: 0 2022-01-11T14:45:29.052 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2022-01-11T14:45:29.048+0000 7f3db8aa25c0 -1 rgw dbstore: shutting down 2022-01-11T14:45:29.052 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:45:29.048+0000 7f9c5a2a2700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster c2 -i x (PID: 31786) UID: 0 2022-01-11T14:45:29.052 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2022-01-11T14:45:29.049+0000 7f9c5a2a2700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2022-01-11T14:45:29.392 INFO:tasks.ceph.c1.osd.0:Sent signal 15 2022-01-11T14:45:29.393 INFO:tasks.ceph.c1.osd.1:Sent signal 15 2022-01-11T14:45:29.393 INFO:tasks.ceph.c1.osd.2:Sent signal 15 2022-01-11T14:45:29.394 INFO:tasks.ceph.c1.mon.a:Sent signal 15 2022-01-11T14:45:29.394 INFO:tasks.rgw.c1.client.0:Sent signal 15 2022-01-11T14:45:29.395 INFO:tasks.ceph.c1.mgr.x:Sent signal 15 2022-01-11T14:45:29.395 INFO:tasks.ceph.c1.osd.0.smithi161.stderr:2022-01-11T14:45:29.395+0000 7f8c51cc0700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 31985) UID: 0 2022-01-11T14:45:29.396 INFO:tasks.ceph.c1.osd.0.smithi161.stderr:2022-01-11T14:45:29.395+0000 7f8c51cc0700 -1 osd.0 61 *** Got signal Terminated *** 2022-01-11T14:45:29.396 INFO:tasks.ceph.c1.osd.0.smithi161.stderr:2022-01-11T14:45:29.395+0000 7f8c51cc0700 -1 osd.0 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-01-11T14:45:29.396 INFO:tasks.ceph.c1.osd.1.smithi161.stderr:2022-01-11T14:45:29.396+0000 7f7195dad700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 31990) UID: 0 2022-01-11T14:45:29.397 INFO:tasks.ceph.c1.osd.1.smithi161.stderr:2022-01-11T14:45:29.396+0000 7f7195dad700 -1 osd.1 61 *** Got signal Terminated *** 2022-01-11T14:45:29.397 INFO:tasks.ceph.c1.osd.1.smithi161.stderr:2022-01-11T14:45:29.396+0000 7f7195dad700 -1 osd.1 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-01-11T14:45:29.397 INFO:tasks.ceph.c1.osd.2.smithi161.stderr:2022-01-11T14:45:29.396+0000 7f03801d5700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 31993) UID: 0 2022-01-11T14:45:29.398 INFO:tasks.ceph.c1.osd.2.smithi161.stderr:2022-01-11T14:45:29.396+0000 7f03801d5700 -1 osd.2 61 *** Got signal Terminated *** 2022-01-11T14:45:29.398 INFO:tasks.ceph.c1.osd.2.smithi161.stderr:2022-01-11T14:45:29.396+0000 7f03801d5700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-01-11T14:45:29.399 INFO:tasks.ceph.c1.mon.a.smithi161.stderr:2022-01-11T14:45:29.397+0000 7f0646403700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 31654) UID: 0 2022-01-11T14:45:29.399 INFO:tasks.ceph.c1.mon.a.smithi161.stderr:2022-01-11T14:45:29.397+0000 7f0646403700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-01-11T14:45:29.400 INFO:tasks.rgw.c1.client.0.smithi161.stdout:2022-01-11T14:45:29.397+0000 7f9bf8adf700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 37133) UID: 0 2022-01-11T14:45:29.400 INFO:tasks.rgw.c1.client.0.smithi161.stdout:2022-01-11T14:45:29.397+0000 7f9c099675c0 -1 rgw dbstore: shutting down 2022-01-11T14:48:38.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... rgw_multi.tests.test_set_bucket_policy ... ERROR 2022-01-11T14:48:38.153 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-51 2022-01-11T14:58:11.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ERROR 2022-01-11T14:58:11.412 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-52 2022-01-11T15:06:50.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ERROR 2022-01-11T15:06:50.620 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-53 2022-01-11T15:15:48.976 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ERROR 2022-01-11T15:15:49.003 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-54 2022-01-11T15:24:36.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ERROR 2022-01-11T15:24:36.988 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-55 2022-01-11T15:33:48.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ERROR 2022-01-11T15:33:48.174 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=epmlcl-56 2022-01-11T15:42:37.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ERROR 2022-01-11T15:42:37.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T15:42:37.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T15:42:37.241 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T15:42:37.294 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T15:42:37.294 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T15:47:37.296 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T15:47:37.296+0000 7fd9ede60500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T15:52:37.297 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T15:52:37.297+0000 7fd9ede60500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T15:57:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T15:57:37.298+0000 7fd9ede60500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T16:02:37.298 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T16:02:37.299+0000 7fd9ede60500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T16:07:37.299 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T16:07:37.300+0000 7fd9ede60500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T16:12:37.300 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T16:12:37.300+0000 7fd9ede60500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T16:17:37.300 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T16:17:37.301+0000 7fd9ede60500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T16:22:37.301 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T16:22:37.301+0000 7fd9ede60500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T16:27:37.301 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T16:27:37.302+0000 7fd9ede60500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T16:32:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-11T16:32:37.341 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T16:32:37.302+0000 7fd9ede60500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T16:32:37.341 INFO:teuthology.orchestra.run.smithi161.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-01-11T16:32:37.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2022-01-11T16:32:37.344 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2022-01-11T16:32:37.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T16:32:37.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T16:32:37.353 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T16:32:37.406 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T16:32:37.406 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T16:37:37.408 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T16:37:37.409+0000 7f1c13072500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T16:42:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T16:42:37.409+0000 7f1c13072500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T16:47:37.409 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T16:47:37.410+0000 7f1c13072500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T16:52:37.410 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T16:52:37.411+0000 7f1c13072500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T16:57:37.412 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T16:57:37.411+0000 7f1c13072500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T17:02:37.411 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T17:02:37.412+0000 7f1c13072500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T17:07:37.411 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T17:07:37.412+0000 7f1c13072500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T17:12:37.412 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T17:12:37.412+0000 7f1c13072500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T17:17:37.413 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T17:17:37.412+0000 7f1c13072500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T17:22:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-11T17:22:37.417 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T17:22:37.414+0000 7f1c13072500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T17:22:37.418 INFO:teuthology.orchestra.run.smithi161.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-01-11T17:22:37.419 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... FAIL 2022-01-11T17:22:37.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T17:22:37.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T17:22:37.422 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T17:22:37.473 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T17:22:37.473 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T17:27:37.476 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T17:27:37.475+0000 7f6726e09500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T17:32:37.476 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T17:32:37.475+0000 7f6726e09500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T17:37:37.477 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T17:37:37.475+0000 7f6726e09500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T17:42:37.478 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T17:42:37.476+0000 7f6726e09500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T17:47:37.479 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T17:47:37.476+0000 7f6726e09500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T17:52:37.479 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T17:52:37.478+0000 7f6726e09500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T17:57:37.480 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T17:57:37.478+0000 7f6726e09500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T18:02:37.481 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T18:02:37.479+0000 7f6726e09500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T18:07:37.482 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T18:07:37.480+0000 7f6726e09500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T18:12:37.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-11T18:12:37.496 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T18:12:37.480+0000 7f6726e09500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T18:12:37.497 INFO:teuthology.orchestra.run.smithi161.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-01-11T18:12:37.499 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... FAIL 2022-01-11T18:12:37.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T18:12:37.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T18:12:37.503 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T18:12:37.554 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T18:12:37.555 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T18:17:37.557 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T18:17:37.556+0000 7f1c44380500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T18:22:37.558 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T18:22:37.555+0000 7f1c44380500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T18:27:37.558 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T18:27:37.556+0000 7f1c44380500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T18:32:37.559 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T18:32:37.556+0000 7f1c44380500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T18:37:37.559 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T18:37:37.558+0000 7f1c44380500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T18:42:37.560 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T18:42:37.558+0000 7f1c44380500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T18:47:37.566 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T18:47:37.558+0000 7f1c44380500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T18:52:37.560 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T18:52:37.559+0000 7f1c44380500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T18:57:37.560 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T18:57:37.559+0000 7f1c44380500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T19:02:37.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-11T19:02:37.598 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T19:02:37.559+0000 7f1c44380500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T19:02:37.598 INFO:teuthology.orchestra.run.smithi161.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-01-11T19:02:37.600 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... FAIL 2022-01-11T19:02:37.601 INFO:tasks.rgw_multisite_tests:test s3 notification filter on master ... ok 2022-01-11T19:02:37.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_object_timing ... ok 2022-01-11T19:02:37.602 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2022-01-11T19:02:37.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T19:02:37.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T19:02:37.604 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T19:02:37.656 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T19:02:37.657 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T19:07:37.658 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T19:07:37.657+0000 7ff63554c500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T19:12:37.657 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T19:12:37.657+0000 7ff63554c500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T19:17:37.658 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T19:17:37.658+0000 7ff63554c500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T19:22:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T19:22:37.657+0000 7ff63554c500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T19:27:37.659 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T19:27:37.658+0000 7ff63554c500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T19:32:37.661 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T19:32:37.659+0000 7ff63554c500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T19:37:37.661 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T19:37:37.659+0000 7ff63554c500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T19:42:37.661 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T19:42:37.660+0000 7ff63554c500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T19:47:37.662 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T19:47:37.660+0000 7ff63554c500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T19:52:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-11T19:52:37.676 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T19:52:37.662+0000 7ff63554c500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T19:52:37.677 INFO:teuthology.orchestra.run.smithi161.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-01-11T19:52:37.679 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... FAIL 2022-01-11T19:52:37.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T19:52:37.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T19:52:37.683 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T19:52:37.735 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T19:52:37.736 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T19:57:37.738 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T19:57:37.738+0000 7ffa3ee98500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T20:02:37.739 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T20:02:37.738+0000 7ffa3ee98500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T20:07:37.739 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T20:07:37.739+0000 7ffa3ee98500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T20:12:37.740 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T20:12:37.739+0000 7ffa3ee98500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T20:17:37.740 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T20:17:37.740+0000 7ffa3ee98500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T20:22:37.740 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T20:22:37.739+0000 7ffa3ee98500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T20:27:37.741 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T20:27:37.740+0000 7ffa3ee98500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T20:32:37.742 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T20:32:37.741+0000 7ffa3ee98500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T20:37:37.743 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T20:37:37.741+0000 7ffa3ee98500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T20:42:37.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-11T20:42:37.794 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T20:42:37.743+0000 7ffa3ee98500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T20:42:37.795 INFO:teuthology.orchestra.run.smithi161.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-01-11T20:42:37.796 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... FAIL 2022-01-11T20:42:37.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T20:42:37.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T20:42:37.802 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T20:42:37.876 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T20:42:37.877 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T20:47:37.879 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T20:47:37.878+0000 7faf6815e500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T20:52:37.880 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T20:52:37.879+0000 7faf6815e500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T20:57:37.880 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T20:57:37.879+0000 7faf6815e500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T21:02:37.881 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T21:02:37.880+0000 7faf6815e500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T21:07:37.882 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T21:07:37.880+0000 7faf6815e500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T21:12:37.882 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T21:12:37.881+0000 7faf6815e500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T21:17:37.883 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T21:17:37.882+0000 7faf6815e500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T21:22:37.884 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T21:22:37.882+0000 7faf6815e500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T21:27:37.883 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T21:27:37.882+0000 7faf6815e500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T21:32:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-11T21:32:37.887 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T21:32:37.883+0000 7faf6815e500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T21:32:37.887 INFO:teuthology.orchestra.run.smithi161.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-01-11T21:32:37.889 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... FAIL 2022-01-11T21:32:37.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T21:32:37.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T21:32:37.891 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T21:32:37.975 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T21:32:37.976 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T21:37:37.978 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T21:37:37.977+0000 7f0367476500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T21:42:37.979 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T21:42:37.978+0000 7f0367476500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T21:47:37.980 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T21:47:37.979+0000 7f0367476500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T21:52:37.982 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T21:52:37.979+0000 7f0367476500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T21:57:37.981 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T21:57:37.980+0000 7f0367476500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T22:02:37.982 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T22:02:37.981+0000 7f0367476500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T22:07:37.981 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T22:07:37.981+0000 7f0367476500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T22:12:37.982 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T22:12:37.982+0000 7f0367476500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T22:17:37.983 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T22:17:37.983+0000 7f0367476500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T22:22:37.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-11T22:22:38.002 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T22:22:37.983+0000 7f0367476500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T22:22:38.002 INFO:teuthology.orchestra.run.smithi161.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-01-11T22:22:38.004 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... FAIL 2022-01-11T22:22:38.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T22:22:38.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T22:22:38.007 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T22:22:38.061 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T22:22:38.062 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T22:27:38.064 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T22:27:38.063+0000 7fd513019500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T22:32:38.065 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T22:32:38.063+0000 7fd513019500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T22:37:38.066 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T22:37:38.065+0000 7fd513019500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T22:42:38.066 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T22:42:38.064+0000 7fd513019500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T22:47:38.069 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T22:47:38.065+0000 7fd513019500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T22:52:38.067 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T22:52:38.066+0000 7fd513019500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T22:57:38.067 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T22:57:38.066+0000 7fd513019500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T23:02:38.068 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T23:02:38.066+0000 7fd513019500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T23:07:38.068 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T23:07:38.067+0000 7fd513019500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T23:12:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-11T23:12:38.072 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T23:12:38.068+0000 7fd513019500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T23:12:38.072 INFO:teuthology.orchestra.run.smithi161.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-01-11T23:12:38.073 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... FAIL 2022-01-11T23:12:38.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T23:12:38.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-11T23:12:38.076 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-11T23:12:38.128 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-11T23:12:38.128 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-11T23:17:38.131 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T23:17:38.130+0000 7f38a84d3500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T23:22:38.131 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T23:22:38.130+0000 7f38a84d3500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T23:27:38.131 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T23:27:38.131+0000 7f38a84d3500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T23:32:38.132 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T23:32:38.131+0000 7f38a84d3500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T23:37:38.133 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T23:37:38.132+0000 7f38a84d3500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T23:42:38.133 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T23:42:38.133+0000 7f38a84d3500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T23:47:38.134 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T23:47:38.133+0000 7f38a84d3500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T23:52:38.134 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T23:52:38.134+0000 7f38a84d3500 0 monclient(hunting): authenticate timed out after 300 2022-01-11T23:57:38.135 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-11T23:57:38.135+0000 7f38a84d3500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T00:02:38.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-12T00:02:38.139 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T00:02:38.136+0000 7f38a84d3500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T00:02:38.140 INFO:teuthology.orchestra.run.smithi161.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-01-12T00:02:38.141 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... FAIL 2022-01-12T00:02:38.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-12T00:02:38.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-12T00:02:38.145 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-12T00:02:38.195 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-12T00:02:38.196 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-12T00:07:38.199 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T00:07:38.199+0000 7f00acd0d500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T00:12:38.199 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T00:12:38.199+0000 7f00acd0d500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T00:17:38.200 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T00:17:38.201+0000 7f00acd0d500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T00:22:38.201 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T00:22:38.200+0000 7f00acd0d500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T00:27:38.201 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T00:27:38.202+0000 7f00acd0d500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T00:32:38.202 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T00:32:38.202+0000 7f00acd0d500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T00:37:38.203 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T00:37:38.202+0000 7f00acd0d500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T00:42:38.202 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T00:42:38.202+0000 7f00acd0d500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T00:47:38.203 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T00:47:38.204+0000 7f00acd0d500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T00:52:38.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-12T00:52:38.207 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T00:52:38.204+0000 7f00acd0d500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T00:52:38.207 INFO:teuthology.orchestra.run.smithi161.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-01-12T00:52:38.209 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... FAIL 2022-01-12T00:52:38.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-12T00:52:38.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-12T00:52:38.211 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-12T00:52:38.262 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-12T00:52:38.262 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-12T00:57:38.266 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T00:57:38.265+0000 7fdcbe634500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T01:02:38.266 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T01:02:38.265+0000 7fdcbe634500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T01:07:38.268 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T01:07:38.266+0000 7fdcbe634500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T01:12:38.267 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T01:12:38.267+0000 7fdcbe634500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T01:17:38.270 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T01:17:38.267+0000 7fdcbe634500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T01:22:38.270 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T01:22:38.267+0000 7fdcbe634500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T01:27:38.271 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T01:27:38.268+0000 7fdcbe634500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T01:32:38.270 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T01:32:38.269+0000 7fdcbe634500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T01:37:38.270 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T01:37:38.269+0000 7fdcbe634500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T01:42:38.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-01-12T01:42:38.275 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T01:42:38.269+0000 7fdcbe634500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T01:42:38.275 INFO:teuthology.orchestra.run.smithi161.stderr:failed to fetch mon config (--no-mon-config to skip) 2022-01-12T01:42:38.276 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... FAIL 2022-01-12T01:42:38.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-12T01:42:38.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2022-01-12T01:42:38.279 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2022-01-12T01:42:38.327 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setuser ceph since I am not root 2022-01-12T01:42:38.328 INFO:teuthology.orchestra.run.smithi161.stderr:ignoring --setgroup ceph since I am not root 2022-01-12T01:47:38.330 INFO:teuthology.orchestra.run.smithi161.stderr:2022-01-12T01:47:38.329+0000 7fd047446500 0 monclient(hunting): authenticate timed out after 300 2022-01-12T01:48:56.490 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-01-12T01:48:56.492 DEBUG:teuthology.task.console_log:Killing console logger for smithi161 2022-01-12T01:48:56.493 DEBUG:teuthology.task.console_log:Killing console logger for smithi172 2022-01-12T01:48:56.493 DEBUG:teuthology.exit:Finished running handlers