2020-11-09T12:20:15.588 INFO:root:teuthology version: 1.0.0-41b569f 2020-11-09T12:20:15.589 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --name ofriedma-2020-11-09_09:43:03-rgw-wip-ofriedma-prefetch-range-i21041-distro-basic-smithi --archive /home/teuthworker/archive/ofriedma-2020-11-09_09:43:03-rgw-wip-ofriedma-prefetch-range-i21041-distro-basic-smithi/5605479 --lock --block --owner scheduled_ofriedma@teuthology --verbose -- /tmp/teuthology-worker.5mam7rvp.tmp 2020-11-09T12:20:15.620 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-09T12:20:15.678 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/ofriedma-2020-11-09_09:43:03-rgw-wip-ofriedma-prefetch-range-i21041-distro-basic-smithi/5605479 branch: wip-ofriedma-prefetch-range-i21041 description: rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: ofriedma@redhat.com first_in_suite: false job_id: '5605479' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: ofriedma-2020-11-09_09:43:03-rgw-wip-ofriedma-prefetch-range-i21041-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-ofriedma-prefetch-range-i21041 ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: afece648f8ccac358b59a7b305a02d8c55dc908d wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: afece648f8ccac358b59a7b305a02d8c55dc908d rgw: compression type: random frontend: civetweb rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-ofriedma-prefetch-range-i21041 sha1: afece648f8ccac358b59a7b305a02d8c55dc908d owner: scheduled_ofriedma@teuthology priority: 101 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: afece648f8ccac358b59a7b305a02d8c55dc908d sleep_before_teardown: 0 suite: rgw suite_branch: wip-ofriedma-prefetch-range-i21041 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-ofriedma-prefetch-range-i21041/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: afece648f8ccac358b59a7b305a02d8c55dc908d tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master timestamp: 2020-11-09_09:43:03 tube: smithi user: ofriedma verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.11564 2020-11-09T12:20:15.679 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-ofriedma-prefetch-range-i21041/qa; will attempt to use it 2020-11-09T12:20:15.679 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-ofriedma-prefetch-range-i21041/qa/tasks 2020-11-09T12:20:15.680 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-09T12:20:15.681 INFO:teuthology.task.internal:Checking packages... 2020-11-09T12:20:15.702 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'afece648f8ccac358b59a7b305a02d8c55dc908d' 2020-11-09T12:20:15.702 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-09T12:20:15.702 INFO:teuthology.packaging:ref: None 2020-11-09T12:20:15.703 INFO:teuthology.packaging:tag: None 2020-11-09T12:20:15.703 INFO:teuthology.packaging:branch: wip-ofriedma-prefetch-range-i21041 2020-11-09T12:20:15.703 INFO:teuthology.packaging:sha1: afece648f8ccac358b59a7b305a02d8c55dc908d 2020-11-09T12:20:15.703 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-ofriedma-prefetch-range-i21041 2020-11-09T12:20:15.868 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-7008-gafece648-1bionic 2020-11-09T12:20:15.869 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-09T12:20:15.879 INFO:teuthology.task.internal:no buildpackages task found 2020-11-09T12:20:15.880 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-09T12:20:15.904 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-09T12:20:15.905 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-09T12:20:15.953 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:20:25.968 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:20:35.988 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:20:46.003 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:20:56.018 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:21:06.035 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:21:16.053 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:21:26.069 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:21:36.086 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:21:46.106 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:21:56.122 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:22:06.139 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:22:16.153 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:22:26.171 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:22:36.187 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:22:46.203 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:22:56.218 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:23:06.237 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:23:16.253 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:23:26.270 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:23:36.286 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:23:46.301 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:23:56.318 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:24:06.334 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:24:16.351 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:24:26.371 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:24:36.395 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:24:46.414 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:24:56.432 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:25:06.447 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:25:16.465 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:25:26.483 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:25:36.500 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:25:46.515 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:25:56.531 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-09T12:26:06.546 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:26:16.563 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:26:26.583 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:26:36.600 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:26:46.620 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:26:56.638 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:27:06.654 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:27:16.671 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:27:26.689 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:27:36.706 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:27:46.722 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-09T12:27:56.742 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_ofriedma@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/ofriedma-2020-11-09_09:43:03-rgw-wip-ofriedma-prefetch-range-i21041-distro-basic-smithi/5605479'} 2020-11-09T12:27:56.766 DEBUG:teuthology.lock.ops:locked ubuntu@smithi164.front.sepia.ceph.com, ubuntu@smithi190.front.sepia.ceph.com 2020-11-09T12:27:56.829 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efe09de1e18>, signals=[15]) 2020-11-09T12:27:57.048 INFO:teuthology.provision.fog.smithi190:Scheduling deploy of ubuntu 18.04 2020-11-09T12:27:57.051 INFO:teuthology.provision.fog.smithi164:Scheduling deploy of ubuntu 18.04 2020-11-09T12:27:57.371 INFO:teuthology.orchestra.console:Power off smithi190 2020-11-09T12:27:57.372 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-09T12:27:57.394 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T12:27:57.415 INFO:teuthology.orchestra.console:Power off smithi164 2020-11-09T12:27:57.416 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-09T12:27:57.438 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T12:27:57.527 INFO:teuthology.orchestra.console:Power off for smithi190 completed 2020-11-09T12:27:57.561 INFO:teuthology.orchestra.console:Power off for smithi164 completed 2020-11-09T12:27:57.629 INFO:teuthology.orchestra.console:Power on smithi190 2020-11-09T12:27:57.630 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-09T12:27:57.652 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T12:27:57.663 INFO:teuthology.orchestra.console:Power on smithi164 2020-11-09T12:27:57.664 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-09T12:27:57.685 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T12:28:01.675 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T12:28:01.708 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T12:28:01.887 INFO:teuthology.orchestra.console:Power on for smithi190 completed 2020-11-09T12:28:01.989 INFO:teuthology.provision.fog.smithi190:Waiting for deploy to finish 2020-11-09T12:28:05.826 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-09T12:28:06.038 INFO:teuthology.orchestra.console:Power on for smithi164 completed 2020-11-09T12:28:06.139 INFO:teuthology.provision.fog.smithi164:Waiting for deploy to finish 2020-11-09T12:31:03.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:31:07.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:31:12.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:31:16.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:31:21.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:31:25.619 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:31:30.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:31:34.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:31:39.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:31:43.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:31:48.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:31:52.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:31:57.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:32:01.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:32:22.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:32:31.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:32:31.256 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-09T12:32:37.259 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:32:37.270 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-09T12:32:39.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:32:39.355 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-09T12:32:43.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:32:43.285 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-09T12:32:45.358 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:32:45.369 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-09T12:32:49.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:32:49.298 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-09T12:32:51.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:32:51.382 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-09T12:32:55.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:32:55.310 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-09T12:32:57.384 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:33:00.423 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-09T12:33:01.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:33:01.409 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-09T12:33:01.410 INFO:teuthology.orchestra.run.smithi190:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-09T12:33:01.759 INFO:teuthology.provision.fog.smithi190:Node is ready 2020-11-09T12:33:01.759 INFO:teuthology.orchestra.run.smithi190:> hostname 2020-11-09T12:33:01.806 INFO:teuthology.orchestra.run.smithi190.stdout:smithi190.front.sepia.ceph.com 2020-11-09T12:33:01.806 INFO:teuthology.orchestra.run.smithi190:> grep smithi190.front.sepia.ceph.com /etc/hosts 2020-11-09T12:33:01.854 INFO:teuthology.orchestra.run.smithi190.stdout:172.21.15.190 smithi190.front.sepia.ceph.com smithi190 2020-11-09T12:33:01.855 INFO:teuthology.orchestra.run.smithi190:> sudo hostname smithi190 2020-11-09T12:33:01.974 INFO:teuthology.orchestra.run.smithi190:> sudo sed -i -e 's/smithi190.front.sepia.ceph.com/smithi190/g' /etc/hosts 2020-11-09T12:33:01.989 INFO:teuthology.orchestra.run.smithi190:> sudo sed -i -e 's/172.21.15.190/172.21.15.190/g' /etc/hosts 2020-11-09T12:33:02.048 INFO:teuthology.orchestra.run.smithi190:> sudo sed -i -e 's/smithi190.front.sepia.ceph.com/smithi190/g' /etc/hostname 2020-11-09T12:33:02.103 INFO:teuthology.orchestra.run.smithi190:> sudo hostname smithi190 2020-11-09T12:33:02.156 INFO:teuthology.provision.fog.smithi190:Deploy complete! 2020-11-09T12:33:02.156 DEBUG:teuthology.parallel:result is None 2020-11-09T12:33:06.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:33:06.500 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-09T12:33:06.501 INFO:teuthology.orchestra.run.smithi164:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-09T12:33:06.833 INFO:teuthology.provision.fog.smithi164:Node is ready 2020-11-09T12:33:06.834 INFO:teuthology.orchestra.run.smithi164:> hostname 2020-11-09T12:33:06.881 INFO:teuthology.orchestra.run.smithi164.stdout:smithi164.front.sepia.ceph.com 2020-11-09T12:33:06.881 INFO:teuthology.orchestra.run.smithi164:> grep smithi164.front.sepia.ceph.com /etc/hosts 2020-11-09T12:33:06.930 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.15.164 smithi164.front.sepia.ceph.com smithi164 2020-11-09T12:33:06.930 INFO:teuthology.orchestra.run.smithi164:> sudo hostname smithi164 2020-11-09T12:33:07.050 INFO:teuthology.orchestra.run.smithi164:> sudo sed -i -e 's/smithi164.front.sepia.ceph.com/smithi164/g' /etc/hosts 2020-11-09T12:33:07.062 INFO:teuthology.orchestra.run.smithi164:> sudo sed -i -e 's/172.21.15.164/172.21.15.164/g' /etc/hosts 2020-11-09T12:33:07.118 INFO:teuthology.orchestra.run.smithi164:> sudo sed -i -e 's/smithi164.front.sepia.ceph.com/smithi164/g' /etc/hostname 2020-11-09T12:33:07.173 INFO:teuthology.orchestra.run.smithi164:> sudo hostname smithi164 2020-11-09T12:33:07.229 INFO:teuthology.provision.fog.smithi164:Deploy complete! 2020-11-09T12:33:07.230 DEBUG:teuthology.parallel:result is None 2020-11-09T12:33:07.368 INFO:teuthology.lock.ops:Checking smithi164.front.sepia.ceph.com 2020-11-09T12:33:07.369 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-09T12:33:07.392 INFO:teuthology.lock.ops:Checking smithi190.front.sepia.ceph.com 2020-11-09T12:33:07.392 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-09T12:33:07.407 INFO:teuthology.lock.ops:Updating [smithi164.front.sepia.ceph.com]: set os type and version on server 2020-11-09T12:33:07.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-09T12:33:07.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:33:07.479 INFO:teuthology.orchestra.run.smithi164:> true 2020-11-09T12:33:07.671 INFO:teuthology.orchestra.run.smithi164:> uname -m 2020-11-09T12:33:07.716 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2020-11-09T12:33:07.717 INFO:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2020-11-09T12:33:07.764 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="Ubuntu" 2020-11-09T12:33:07.765 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-09T12:33:07.765 INFO:teuthology.orchestra.run.smithi164.stdout:ID=ubuntu 2020-11-09T12:33:07.765 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE=debian 2020-11-09T12:33:07.765 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-09T12:33:07.765 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="18.04" 2020-11-09T12:33:07.765 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-09T12:33:07.766 INFO:teuthology.orchestra.run.smithi164.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-09T12:33:07.766 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-09T12:33:07.766 INFO:teuthology.orchestra.run.smithi164.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-09T12:33:07.766 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_CODENAME=bionic 2020-11-09T12:33:07.766 INFO:teuthology.orchestra.run.smithi164.stdout:UBUNTU_CODENAME=bionic 2020-11-09T12:33:07.767 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2020-11-09T12:33:07.789 INFO:teuthology.lock.ops:Updating [smithi190.front.sepia.ceph.com]: set os type and version on server 2020-11-09T12:33:07.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-09T12:33:07.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:33:07.865 INFO:teuthology.orchestra.run.smithi190:> true 2020-11-09T12:33:08.107 INFO:teuthology.orchestra.run.smithi190:> uname -m 2020-11-09T12:33:08.154 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2020-11-09T12:33:08.155 INFO:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2020-11-09T12:33:08.202 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="Ubuntu" 2020-11-09T12:33:08.203 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-09T12:33:08.203 INFO:teuthology.orchestra.run.smithi190.stdout:ID=ubuntu 2020-11-09T12:33:08.203 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE=debian 2020-11-09T12:33:08.203 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-09T12:33:08.203 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="18.04" 2020-11-09T12:33:08.203 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-09T12:33:08.204 INFO:teuthology.orchestra.run.smithi190.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-09T12:33:08.204 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-09T12:33:08.204 INFO:teuthology.orchestra.run.smithi190.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-09T12:33:08.204 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_CODENAME=bionic 2020-11-09T12:33:08.204 INFO:teuthology.orchestra.run.smithi190.stdout:UBUNTU_CODENAME=bionic 2020-11-09T12:33:08.205 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2020-11-09T12:33:08.220 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-11-09T12:33:08.254 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi164.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDe4y9371xWFuJuIE04B1Fo1b3ZUH7GU9oATqw2xFl1QfUisADv12XzhKqc0ugtdpxyKvS/gawH3evS0UNqAISY3XCL6/tDGLyFGYGSKt64jJDB8sDZjAM70CW+2a4LHdnUOwU/1WgrN3UJt1bvxqBJx1JKxzf1tLXpbNS1ddAYdpFO8JfVRWDzAJE9A1sMDxwKLol7ugg8NcoXhgwIZy3c1njFGKsw2qLUDFlLX7uFl0ZcfXHSi/fbT75rz38IpYMnIBWsvJ7p6eeOZ2bggJEjqnxplzB53wy0tj0z++vsj1B7lAcm8poGBCEkckeoGJBs5USEpY/SJrhggEA4SUv5 smithi190.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQeVG89hdn6N416X1bJbSagXnM97175QAWzlHiMK4psAkbs3KVf9m0VKtccgsoREVvt7MGw3QTez9R2g4MBvQuto2PHUqUZqq5V4Pm6tBV4JnivVwTLThK/9yXXQDKyHaDV5tQ0q0KO9QtVod93+J+ksEm+zhq8BJM4l/gn08hIM/1adp4g8wOP1AjtZ+CdHm7PGUdDKTmrYpNx5Jw9dyGMKrTAcnq9sgWO7i1YlbXRIj1wTj3N6+X2DmoiI7jYLixC7D5IaSvwZ4BrnWqAKeIoG3Cbp2aQK01TLELHgAZnc9N0J/xD8rwZ2oLZTc8dXgzTkXqzStjS+v0erO1zVb5 2020-11-09T12:33:08.255 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-09T12:33:08.308 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-09T12:33:08.313 INFO:teuthology.task.internal:Saving configuration 2020-11-09T12:33:08.328 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-09T12:33:08.346 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-09T12:33:08.364 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-09 12:27:56.753209', 'description': '/home/teuthworker/archive/ofriedma-2020-11-09_09:43:03-rgw-wip-ofriedma-prefetch-range-i21041-distro-basic-smithi/5605479', 'locked_by': 'scheduled_ofriedma@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP+DjVHtmmykQ7wwFg+MMPbV0Tz09Pf2dLm6nMg5QjpH', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi164.front.sepia.ceph.com'} 2020-11-09T12:33:08.380 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-09 12:27:56.753233', 'description': '/home/teuthworker/archive/ofriedma-2020-11-09_09:43:03-rgw-wip-ofriedma-prefetch-range-i21041-distro-basic-smithi/5605479', 'locked_by': 'scheduled_ofriedma@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGEOt65eJAyaTG9e3SoE+o5UtdUuesfeZxOgzFYY8N50', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi190.front.sepia.ceph.com'} 2020-11-09T12:33:08.380 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-09T12:33:08.392 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-11-09T12:33:08.393 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-11-09T12:33:08.393 INFO:teuthology.run_tasks:Running task console_log... 2020-11-09T12:33:08.466 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efe09e3ff28>, signals=[15]) 2020-11-09T12:33:08.467 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-09T12:33:08.477 INFO:teuthology.task.internal:Opening connections... 2020-11-09T12:33:08.478 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2020-11-09T12:33:08.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:33:08.550 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2020-11-09T12:33:08.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:33:08.625 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-09T12:33:08.636 INFO:teuthology.orchestra.run.smithi164:> uname -m 2020-11-09T12:33:08.722 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2020-11-09T12:33:08.722 INFO:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2020-11-09T12:33:08.770 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="Ubuntu" 2020-11-09T12:33:08.771 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-09T12:33:08.771 INFO:teuthology.orchestra.run.smithi164.stdout:ID=ubuntu 2020-11-09T12:33:08.771 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE=debian 2020-11-09T12:33:08.771 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-09T12:33:08.771 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="18.04" 2020-11-09T12:33:08.772 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-09T12:33:08.772 INFO:teuthology.orchestra.run.smithi164.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-09T12:33:08.772 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-09T12:33:08.772 INFO:teuthology.orchestra.run.smithi164.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-09T12:33:08.772 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_CODENAME=bionic 2020-11-09T12:33:08.772 INFO:teuthology.orchestra.run.smithi164.stdout:UBUNTU_CODENAME=bionic 2020-11-09T12:33:08.773 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2020-11-09T12:33:08.790 INFO:teuthology.orchestra.run.smithi190:> uname -m 2020-11-09T12:33:08.814 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2020-11-09T12:33:08.814 INFO:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2020-11-09T12:33:08.862 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="Ubuntu" 2020-11-09T12:33:08.862 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-09T12:33:08.863 INFO:teuthology.orchestra.run.smithi190.stdout:ID=ubuntu 2020-11-09T12:33:08.863 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE=debian 2020-11-09T12:33:08.863 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-09T12:33:08.863 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="18.04" 2020-11-09T12:33:08.863 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-09T12:33:08.863 INFO:teuthology.orchestra.run.smithi190.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-09T12:33:08.864 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-09T12:33:08.864 INFO:teuthology.orchestra.run.smithi190.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-09T12:33:08.864 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_CODENAME=bionic 2020-11-09T12:33:08.864 INFO:teuthology.orchestra.run.smithi190.stdout:UBUNTU_CODENAME=bionic 2020-11-09T12:33:08.864 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2020-11-09T12:33:08.880 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-09T12:33:08.933 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-09T12:33:08.944 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-09T12:33:08.945 INFO:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2020-11-09T12:33:08.948 INFO:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2020-11-09T12:33:08.954 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-09T12:33:08.965 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-09T12:33:08.966 INFO:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2020-11-09T12:33:08.997 INFO:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2020-11-09T12:33:09.006 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-09T12:33:09.048 INFO:teuthology.run_tasks:Running task kernel... 2020-11-09T12:33:09.064 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-11-09T12:33:09.064 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-11-09T12:33:09.065 INFO:teuthology.orchestra.run.smithi164:> uname -r 2020-11-09T12:33:09.071 INFO:teuthology.orchestra.run.smithi164.stdout:4.15.0-122-generic 2020-11-09T12:33:09.071 INFO:teuthology.task.kernel:Running kernel on smithi164: 4.15.0-122-generic 2020-11-09T12:33:09.071 INFO:teuthology.orchestra.run.smithi164:> sudo apt-get clean 2020-11-09T12:33:09.231 INFO:teuthology.orchestra.run.smithi164:> sudo apt-get update 2020-11-09T12:33:09.577 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-09T12:33:09.578 INFO:teuthology.orchestra.run.smithi164.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-09T12:33:09.642 INFO:teuthology.orchestra.run.smithi164.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-09T12:33:09.915 INFO:teuthology.orchestra.run.smithi164.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-09T12:33:10.286 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 252 kB in 1s (336 kB/s) 2020-11-09T12:33:11.910 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2020-11-09T12:33:11.959 INFO:teuthology.orchestra.run.smithi164:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-09T12:33:12.046 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2020-11-09T12:33:12.299 INFO:teuthology.orchestra.run.smithi164.stdout:Building dependency tree... 2020-11-09T12:33:12.301 INFO:teuthology.orchestra.run.smithi164.stdout:Reading state information... 2020-11-09T12:33:12.379 INFO:teuthology.orchestra.run.smithi164.stderr:E: Unable to locate package linux-image-current-generic 2020-11-09T12:33:12.381 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-09T12:33:12.382 INFO:teuthology.orchestra.run.smithi164:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-09T12:33:12.473 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2020-11-09T12:33:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:Building dependency tree... 2020-11-09T12:33:12.723 INFO:teuthology.orchestra.run.smithi164.stdout:Reading state information... 2020-11-09T12:33:12.923 INFO:teuthology.orchestra.run.smithi164.stdout:linux-image-generic is already the newest version (4.15.0.122.109). 2020-11-09T12:33:12.924 INFO:teuthology.orchestra.run.smithi164.stdout:linux-image-generic set to manually installed. 2020-11-09T12:33:12.924 INFO:teuthology.orchestra.run.smithi164.stdout:0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 2020-11-09T12:33:12.925 INFO:teuthology.orchestra.run.smithi164:> dpkg -s linux-image-generic 2020-11-09T12:33:12.949 INFO:teuthology.orchestra.run.smithi164.stdout:Package: linux-image-generic 2020-11-09T12:33:12.950 INFO:teuthology.orchestra.run.smithi164.stdout:Status: install ok installed 2020-11-09T12:33:12.950 INFO:teuthology.orchestra.run.smithi164.stdout:Priority: optional 2020-11-09T12:33:12.950 INFO:teuthology.orchestra.run.smithi164.stdout:Section: kernel 2020-11-09T12:33:12.950 INFO:teuthology.orchestra.run.smithi164.stdout:Installed-Size: 16 2020-11-09T12:33:12.950 INFO:teuthology.orchestra.run.smithi164.stdout:Maintainer: Ubuntu Kernel Team 2020-11-09T12:33:12.950 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: amd64 2020-11-09T12:33:12.951 INFO:teuthology.orchestra.run.smithi164.stdout:Source: linux-meta 2020-11-09T12:33:12.951 INFO:teuthology.orchestra.run.smithi164.stdout:Version: 4.15.0.122.109 2020-11-09T12:33:12.951 INFO:teuthology.orchestra.run.smithi164.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-09T12:33:12.951 INFO:teuthology.orchestra.run.smithi164.stdout:Depends: linux-image-4.15.0-122-generic, linux-modules-extra-4.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-09T12:33:12.951 INFO:teuthology.orchestra.run.smithi164.stdout:Recommends: thermald 2020-11-09T12:33:12.951 INFO:teuthology.orchestra.run.smithi164.stdout:Description: Generic Linux kernel image 2020-11-09T12:33:12.952 INFO:teuthology.orchestra.run.smithi164.stdout: This package will always depend on the latest generic kernel image 2020-11-09T12:33:12.952 INFO:teuthology.orchestra.run.smithi164.stdout: available. 2020-11-09T12:33:12.952 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-09T12:33:12.953 INFO:teuthology.orchestra.run.smithi190:> uname -r 2020-11-09T12:33:12.959 INFO:teuthology.orchestra.run.smithi190.stdout:4.15.0-122-generic 2020-11-09T12:33:12.960 INFO:teuthology.task.kernel:Running kernel on smithi190: 4.15.0-122-generic 2020-11-09T12:33:12.960 INFO:teuthology.orchestra.run.smithi190:> sudo apt-get clean 2020-11-09T12:33:13.113 INFO:teuthology.orchestra.run.smithi190:> sudo apt-get update 2020-11-09T12:33:13.461 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-09T12:33:13.461 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-09T12:33:13.526 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-09T12:33:13.821 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-09T12:33:14.164 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 252 kB in 1s (325 kB/s) 2020-11-09T12:33:15.775 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2020-11-09T12:33:15.823 INFO:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-09T12:33:15.903 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2020-11-09T12:33:16.153 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2020-11-09T12:33:16.154 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2020-11-09T12:33:16.232 INFO:teuthology.orchestra.run.smithi190.stderr:E: Unable to locate package linux-image-current-generic 2020-11-09T12:33:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-09T12:33:16.234 INFO:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-09T12:33:16.310 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2020-11-09T12:33:16.567 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2020-11-09T12:33:16.568 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2020-11-09T12:33:16.775 INFO:teuthology.orchestra.run.smithi190.stdout:linux-image-generic is already the newest version (4.15.0.122.109). 2020-11-09T12:33:16.775 INFO:teuthology.orchestra.run.smithi190.stdout:linux-image-generic set to manually installed. 2020-11-09T12:33:16.775 INFO:teuthology.orchestra.run.smithi190.stdout:0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 2020-11-09T12:33:16.776 INFO:teuthology.orchestra.run.smithi190:> dpkg -s linux-image-generic 2020-11-09T12:33:16.800 INFO:teuthology.orchestra.run.smithi190.stdout:Package: linux-image-generic 2020-11-09T12:33:16.800 INFO:teuthology.orchestra.run.smithi190.stdout:Status: install ok installed 2020-11-09T12:33:16.800 INFO:teuthology.orchestra.run.smithi190.stdout:Priority: optional 2020-11-09T12:33:16.800 INFO:teuthology.orchestra.run.smithi190.stdout:Section: kernel 2020-11-09T12:33:16.801 INFO:teuthology.orchestra.run.smithi190.stdout:Installed-Size: 16 2020-11-09T12:33:16.801 INFO:teuthology.orchestra.run.smithi190.stdout:Maintainer: Ubuntu Kernel Team 2020-11-09T12:33:16.801 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: amd64 2020-11-09T12:33:16.801 INFO:teuthology.orchestra.run.smithi190.stdout:Source: linux-meta 2020-11-09T12:33:16.801 INFO:teuthology.orchestra.run.smithi190.stdout:Version: 4.15.0.122.109 2020-11-09T12:33:16.801 INFO:teuthology.orchestra.run.smithi190.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-09T12:33:16.801 INFO:teuthology.orchestra.run.smithi190.stdout:Depends: linux-image-4.15.0-122-generic, linux-modules-extra-4.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-09T12:33:16.802 INFO:teuthology.orchestra.run.smithi190.stdout:Recommends: thermald 2020-11-09T12:33:16.802 INFO:teuthology.orchestra.run.smithi190.stdout:Description: Generic Linux kernel image 2020-11-09T12:33:16.802 INFO:teuthology.orchestra.run.smithi190.stdout: This package will always depend on the latest generic kernel image 2020-11-09T12:33:16.802 INFO:teuthology.orchestra.run.smithi190.stdout: available. 2020-11-09T12:33:16.802 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-09T12:33:16.803 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-11-09T12:33:16.803 INFO:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-09T12:33:16.840 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2020-11-09T12:33:16.874 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-11-09T12:33:16.875 INFO:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-09T12:33:16.896 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2020-11-09T12:33:16.915 INFO:teuthology.run_tasks:Running task internal.base... 2020-11-09T12:33:16.956 INFO:teuthology.task.internal:Creating test directory... 2020-11-09T12:33:16.957 INFO:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-09T12:33:16.964 INFO:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-09T12:33:16.974 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-11-09T12:33:16.988 INFO:teuthology.run_tasks:Running task internal.archive... 2020-11-09T12:33:17.025 INFO:teuthology.task.internal:Creating archive directory... 2020-11-09T12:33:17.026 INFO:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-09T12:33:17.029 INFO:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-09T12:33:17.041 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-11-09T12:33:17.047 INFO:teuthology.task.internal:Enabling coredump saving... 2020-11-09T12:33:17.047 INFO:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-09T12:33:17.080 INFO:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-09T12:33:17.097 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-09T12:33:17.107 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-09T12:33:17.123 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-09T12:33:17.133 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-09T12:33:17.135 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-11-09T12:33:17.143 INFO:teuthology.task.internal:Configuring sudo... 2020-11-09T12:33:17.144 INFO:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-09T12:33:17.152 INFO:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-09T12:33:17.190 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-11-09T12:33:17.199 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-11-09T12:33:17.200 INFO:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-09T12:33:17.208 INFO:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-09T12:33:17.238 INFO:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-09T12:33:17.260 INFO:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-09T12:33:17.312 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:33:17.313 INFO:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-09T12:33:17.358 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-09T12:33:17.376 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:33:17.377 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:33:17.377 INFO:teuthology.orchestra.run.smithi164.stderr:152 bytes copied, 3.3675e-05 s, 4.5 MB/s 2020-11-09T12:33:17.378 INFO:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-09T12:33:17.385 INFO:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-09T12:33:17.434 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:33:17.435 INFO:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-09T12:33:17.480 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-09T12:33:17.496 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:33:17.497 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:33:17.497 INFO:teuthology.orchestra.run.smithi190.stderr:152 bytes copied, 3.6484e-05 s, 4.2 MB/s 2020-11-09T12:33:17.498 INFO:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2020-11-09T12:33:17.501 INFO:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2020-11-09T12:33:17.579 INFO:teuthology.run_tasks:Running task internal.timer... 2020-11-09T12:33:17.586 INFO:teuthology.task.internal:Starting timer... 2020-11-09T12:33:17.587 INFO:teuthology.run_tasks:Running task pcp... 2020-11-09T12:33:17.661 INFO:teuthology.run_tasks:Running task selinux... 2020-11-09T12:33:17.672 DEBUG:teuthology.task.selinux:Excluding smithi164: OS 'ubuntu' does not support SELinux 2020-11-09T12:33:17.673 DEBUG:teuthology.task.selinux:Excluding smithi190: OS 'ubuntu' does not support SELinux 2020-11-09T12:33:17.673 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-11-09T12:33:17.673 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-11-09T12:33:17.673 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-11-09T12:33:17.674 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-11-09T12:33:17.683 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-11-09T12:33:17.689 INFO:teuthology.repo_utils:Fetching master from origin 2020-11-09T12:33:17.739 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-11-09T12:33:17.752 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-11-09T12:33:17.753 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi164.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-11-09T12:33:18.071 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-11-09T12:33:18.866 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-11-09T12:33:21.013 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-11-09T12:33:21.033 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:21 +0000 (0:00:00.067) 0:00:00.067 ******* 2020-11-09T12:33:21.048 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:21 +0000 (0:00:00.015) 0:00:00.082 ******* 2020-11-09T12:33:22.697 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi190.front.sepia.ceph.com] 2020-11-09T12:33:22.860 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:22 +0000 (0:00:01.811) 0:00:01.894 ******* 2020-11-09T12:33:22.871 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] 2020-11-09T12:33:22.943 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:22 +0000 (0:00:00.082) 0:00:01.977 ******* 2020-11-09T12:33:23.134 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-11-09T12:33:23.136 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-09T12:33:23.167 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:23 +0000 (0:00:00.224) 0:00:02.201 ******* 2020-11-09T12:33:23.197 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:23 +0000 (0:00:00.029) 0:00:02.231 ******* 2020-11-09T12:33:23.618 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-11-09T12:33:23.620 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} changed: [smithi190.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-09T12:33:23.654 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:23 +0000 (0:00:00.456) 0:00:02.688 ******* 2020-11-09T12:33:23.689 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:23 +0000 (0:00:00.034) 0:00:02.723 ******* 2020-11-09T12:33:23.984 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-11-09T12:33:23.986 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.065603", "end": "2020-11-09 12:33:23.495490", "rc": 0, "start": "2020-11-09 12:33:23.429887", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-11-09T12:33:24.018 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:24 +0000 (0:00:00.328) 0:00:03.052 ******* 2020-11-09T12:33:24.025 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.072791", "end": "2020-11-09 12:33:23.857583", "rc": 0, "start": "2020-11-09 12:33:23.784792", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-11-09T12:33:24.060 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:24 +0000 (0:00:00.042) 0:00:03.094 ******* 2020-11-09T12:33:24.251 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-11-09T12:33:24.253 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T12:33:24.285 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:24 +0000 (0:00:00.225) 0:00:03.319 ******* 2020-11-09T12:33:24.292 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T12:33:24.329 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:24 +0000 (0:00:00.043) 0:00:03.363 ******* 2020-11-09T12:33:24.932 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-11-09T12:33:24.934 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1604925204.2990663-36272883651286/source", "state": "file", "uid": 0} 2020-11-09T12:33:24.969 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:24 +0000 (0:00:00.639) 0:00:04.003 ******* 2020-11-09T12:33:24.977 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1604925204.3429656-73136557109197/source", "state": "file", "uid": 0} 2020-11-09T12:33:25.013 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:25 +0000 (0:00:00.044) 0:00:04.047 ******* 2020-11-09T12:33:25.279 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-11-09T12:33:25.281 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-09T12:33:25.283 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-09T12:33:25.312 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-09T12:33:25.314 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-09T12:33:25.422 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-11-09T12:33:25.428 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-11-09T12:33:25.554 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:25 +0000 (0:00:00.540) 0:00:04.588 ******* 2020-11-09T12:33:25.585 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:25 +0000 (0:00:00.030) 0:00:04.619 ******* 2020-11-09T12:33:26.350 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi190.front.sepia.ceph.com] 2020-11-09T12:33:26.438 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-11-09T12:33:26.439 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:26 +0000 (0:00:00.852) 0:00:05.472 ******* 2020-11-09T12:33:26.802 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-11-09T12:33:26.804 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.165970", "end": "2020-11-09 12:33:26.774719", "rc": 0, "start": "2020-11-09 12:33:26.608749", "stderr": "", "stderr_lines": [], "stdout": "f76e9e239faceca753ffd443c97bd3654ebef2d1", "stdout_lines": ["f76e9e239faceca753ffd443c97bd3654ebef2d1"]} 2020-11-09T12:33:26.849 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:26 +0000 (0:00:00.411) 0:00:05.883 ******* 2020-11-09T12:33:26.994 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-11-09T12:33:26.996 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.017934", "end": "2020-11-09 12:33:26.865363", "failed_when_result": false, "rc": 0, "start": "2020-11-09 12:33:26.847429", "stderr": "", "stderr_lines": [], "stdout": "f76e9e239faceca753ffd443c97bd3654ebef2d1", "stdout_lines": ["f76e9e239faceca753ffd443c97bd3654ebef2d1"]} 2020-11-09T12:33:27.031 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:27 +0000 (0:00:00.181) 0:00:06.065 ******* 2020-11-09T12:33:27.058 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-11-09T12:33:27.060 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-09T12:33:27.091 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:27 +0000 (0:00:00.060) 0:00:06.125 ******* 2020-11-09T12:33:27.118 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-11-09T12:33:27.120 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.161 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:27 +0000 (0:00:00.070) 0:00:06.195 ******* 2020-11-09T12:33:27.190 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-11-09T12:33:27.192 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.237 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:27 +0000 (0:00:00.075) 0:00:06.271 ******* 2020-11-09T12:33:27.270 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-11-09T12:33:27.272 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.307 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:27 +0000 (0:00:00.070) 0:00:06.341 ******* 2020-11-09T12:33:27.318 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-11-09T12:33:27.321 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] 2020-11-09T12:33:27.358 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-11-09T12:33:27.361 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.394 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:27 +0000 (0:00:00.086) 0:00:06.428 ******* 2020-11-09T12:33:27.426 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:27 +0000 (0:00:00.031) 0:00:06.460 ******* 2020-11-09T12:33:27.458 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-11-09T12:33:27.460 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.493 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:27 +0000 (0:00:00.067) 0:00:06.527 ******* 2020-11-09T12:33:27.542 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T12:33:27.544 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.554 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.555 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.567 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.568 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.578 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.580 INFO:teuthology.task.ansible.out: 2020-11-09T12:33:27.590 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.592 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.601 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.603 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.613 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.614 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.624 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.626 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.635 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.637 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.647 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.648 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.658 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.660 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.670 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.680 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.682 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.691 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.694 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.703 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.705 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.714 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.716 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.725 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.736 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.737 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-11-09T12:33:27.737 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.152988", "end": "2020-11-09 12:33:27.710037", "rc": 0, "start": "2020-11-09 12:33:27.557049", "stderr": "", "stderr_lines": [], "stdout": "f76e9e239faceca753ffd443c97bd3654ebef2d1", "stdout_lines": ["f76e9e239faceca753ffd443c97bd3654ebef2d1"]} 2020-11-09T12:33:27.737 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.772 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:27 +0000 (0:00:00.278) 0:00:06.806 ******* 2020-11-09T12:33:27.779 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.780 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.782 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.782 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.783 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@v 2020-11-09T12:33:27.784 INFO:teuthology.task.ansible.out:umrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.795 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.796 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.806 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.817 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.819 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.829 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.830 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.840 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.842 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.852 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.853 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.863 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.865 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.875 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.876 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.876 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.887 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.889 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.898 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.900 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.910 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.911 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.018044", "end": "2020-11-09 12:33:27.431347", "failed_when_result": false, "rc": 0, "start": "2020-11-09 12:33:27.413303", "stderr": "", "stderr_lines": [], "stdout": "f76e9e239faceca753ffd443c97bd3654ebef2d1", "stdout_lines": ["f76e9e239faceca753ffd443c97bd3654ebef2d1"]} 2020-11-09T12:33:27.947 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:27 +0000 (0:00:00.174) 0:00:06.981 ******* 2020-11-09T12:33:27.953 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T12:33:27.955 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.955 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.956 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.956 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehl 2020-11-09T12:33:27.957 INFO:teuthology.task.ansible.out:er", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.969 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.970 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.980 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:27.981 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.015 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:28 +0000 (0:00:00.068) 0:00:07.049 ******* 2020-11-09T12:33:28.022 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.023 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.025 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.025 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.026 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.027 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f9 2020-11-09T12:33:28.029 INFO:teuthology.task.ansible.out:97d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.038 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.040 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T12:33:28.040 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.075 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:28 +0000 (0:00:00.060) 0:00:07.109 ******* 2020-11-09T12:33:28.083 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.084 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.084 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.086 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.086 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.088 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.098 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.100 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.135 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:28 +0000 (0:00:00.060) 0:00:07.169 ******* 2020-11-09T12:33:28.144 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.146 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.146 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.147 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.148 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.148 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.150 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.160 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.162 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} TASK [users : Filter the managed_users list] *********************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.195 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:28 +0000 (0:00:00.059) 0:00:07.229 ******* 2020-11-09T12:33:28.202 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.204 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.206 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.206 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.217 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.219 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.229 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.231 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.266 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:28 +0000 (0:00:00.070) 0:00:07.300 ******* 2020-11-09T12:33:28.273 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.275 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.275 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansib 2020-11-09T12:33:28.278 INFO:teuthology.task.ansible.out:le_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.291 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.324 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:28 +0000 (0:00:00.057) 0:00:07.358 ******* 2020-11-09T12:33:28.356 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:28 +0000 (0:00:00.031) 0:00:07.390 ******* 2020-11-09T12:33:28.375 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T12:33:28.377 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} TASK [users : Create all users without sudo access.] *************************** 2020-11-09T12:33:28.412 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:28 +0000 (0:00:00.055) 0:00:07.446 ******* 2020-11-09T12:33:28.418 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T12:33:28.420 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.420 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr 2020-11-09T12:33:28.422 INFO:teuthology.task.ansible.out:+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.422 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_l 2020-11-09T12:33:28.423 INFO:teuthology.task.ansible.out:oop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.425 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.435 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.436 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-11-09T12:33:28.436 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.437 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.437 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', ' 2020-11-09T12:33:28.438 INFO:teuthology.task.ansible.out:ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.475 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:28 +0000 (0:00:00.063) 0:00:07.509 ******* 2020-11-09T12:33:28.483 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.484 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.486 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.487 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => 2020-11-09T12:33:28.488 INFO:teuthology.task.ansible.out: (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item" 2020-11-09T12:33:28.490 INFO:teuthology.task.ansible.out:, "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.499 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.500 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.501 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.541 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:28 +0000 (0:00:00.065) 0:00:07.575 ******* 2020-11-09T12:33:28.549 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.550 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.551 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.552 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTf 2020-11-09T12:33:28.553 INFO:teuthology.task.ansible.out:DCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.564 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.566 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.599 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:28 +0000 (0:00:00.058) 0:00:07.633 ******* 2020-11-09T12:33:28.606 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.607 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.608 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.609 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.609 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.610 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'vasu', ' 2020-11-09T12:33:28.611 INFO:teuthology.task.ansible.out:ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.621 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.623 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.633 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.634 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.671 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:28 +0000 (0:00:00.071) 0:00:07.705 ******* 2020-11-09T12:33:28.681 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.683 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.683 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.684 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (i 2020-11-09T12:33:28.686 INFO:teuthology.task.ansible.out:tem={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f1 2020-11-09T12:33:28.688 INFO:teuthology.task.ansible.out:8b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.698 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.699 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.733 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:28 +0000 (0:00:00.062) 0:00:07.767 ******* 2020-11-09T12:33:28.741 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.743 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.743 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.744 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.745 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", 2020-11-09T12:33:28.747 INFO:teuthology.task.ansible.out:"ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.757 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.768 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.770 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.780 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T12:33:28.782 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.793 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.795 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.805 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.807 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.817 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.819 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20ef 2020-11-09T12:33:28.821 INFO:teuthology.task.ansible.out:c114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.830 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.832 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.842 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.844 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.854 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.856 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": " 2020-11-09T12:33:28.858 INFO:teuthology.task.ansible.out:lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.868 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T12:33:28.869 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.880 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.881 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.882 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn" 2020-11-09T12:33:28.883 INFO:teuthology.task.ansible.out:: "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.893 INFO:teuthology.task.ansible.out: 2020-11-09T12:33:28.894 INFO:teuthology.task.ansible.out:TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.897 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.907 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T12:33:28.908 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.909 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.909 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => { 2020-11-09T12:33:28.910 INFO:teuthology.task.ansible.out:"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.920 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T12:33:28.922 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.922 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T12:33:28.923 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.924 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.934 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T12:33:28.936 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.946 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T12:33:28.948 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was 2020-11-09T12:33:28.950 INFO:teuthology.task.ansible.out:False"} 2020-11-09T12:33:28.959 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T12:33:28.961 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.961 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.961 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf0560 2020-11-09T12:33:28.963 INFO:teuthology.task.ansible.out:14c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.974 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T12:33:28.976 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.986 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-09T12:33:28.988 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:28.991 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.043 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:29 +0000 (0:00:00.310) 0:00:08.077 ******* 2020-11-09T12:33:29.053 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.055 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.056 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.058 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.058 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.060 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.061 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.063 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.063 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.064 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.115 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:29 +0000 (0:00:00.072) 0:00:08.149 ******* 2020-11-09T12:33:29.146 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-11-09T12:33:29.178 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:29 +0000 (0:00:00.062) 0:00:08.212 ******* 2020-11-09T12:33:29.206 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-11-09T12:33:29.208 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.243 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:29 +0000 (0:00:00.065) 0:00:08.277 ******* 2020-11-09T12:33:29.271 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-11-09T12:33:29.273 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.305 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:29 +0000 (0:00:00.061) 0:00:08.339 ******* 2020-11-09T12:33:29.333 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-09T12:33:29.335 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.367 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:29 +0000 (0:00:00.062) 0:00:08.401 ******* 2020-11-09T12:33:29.396 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-09T12:33:29.398 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.431 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:29 +0000 (0:00:00.064) 0:00:08.465 ******* 2020-11-09T12:33:29.460 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-11-09T12:33:29.462 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.496 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:29 +0000 (0:00:00.064) 0:00:08.530 ******* 2020-11-09T12:33:29.537 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-09T12:33:29.538 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.540 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.550 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.551 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.562 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.563 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.575 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.576 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.576 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.587 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.598 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.599 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.611 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.622 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-11-09T12:33:29.623 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "d026344f7746f1ffdf46ed470dbbe51622ca670c", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-11-09T12:33:29.625 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi164.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.625 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.627 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.665 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:29 +0000 (0:00:00.168) 0:00:08.699 ******* 2020-11-09T12:33:29.674 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.676 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.677 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.677 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.688 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.690 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.700 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi190.front.sepia.ceph.com] => { "msg": "Host smithi190.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-09T12:33:29.703 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi164.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.741 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:29 +0000 (0:00:00.076) 0:00:08.775 ******* 2020-11-09T12:33:29.751 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.753 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.754 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.756 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.757 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.759 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.770 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.771 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.772 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.772 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.814 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:29 +0000 (0:00:00.072) 0:00:08.848 ******* 2020-11-09T12:33:29.826 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi190.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-09T12:33:29.865 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:29 +0000 (0:00:00.051) 0:00:08.899 ******* 2020-11-09T12:33:29.894 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-11-09T12:33:29.895 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.928 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:29 +0000 (0:00:00.062) 0:00:08.962 ******* 2020-11-09T12:33:29.955 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-09T12:33:29.957 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:29.994 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:29 +0000 (0:00:00.066) 0:00:09.028 ******* 2020-11-09T12:33:30.021 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-11-09T12:33:30.023 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.057 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.062) 0:00:09.091 ******* 2020-11-09T12:33:30.085 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-09T12:33:30.086 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.119 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.061) 0:00:09.153 ******* 2020-11-09T12:33:30.138 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-11-09T12:33:30.139 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "d026344f7746f1ffdf46ed470dbbe51622ca670c", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-11-09T12:33:30.174 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.055) 0:00:09.208 ******* 2020-11-09T12:33:30.181 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-09T12:33:30.183 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.218 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.042) 0:00:09.250 ******* 2020-11-09T12:33:30.223 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-11-09T12:33:30.225 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => { "msg": "Host smithi164.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-09T12:33:30.259 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.042) 0:00:09.292 ******* 2020-11-09T12:33:30.266 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-11-09T12:33:30.268 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.311 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.051) 0:00:09.344 ******* 2020-11-09T12:33:30.316 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-11-09T12:33:30.318 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-09T12:33:30.354 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.043) 0:00:09.388 ******* 2020-11-09T12:33:30.361 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-11-09T12:33:30.363 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.396 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.041) 0:00:09.430 ******* 2020-11-09T12:33:30.403 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-11-09T12:33:30.404 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.439 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.042) 0:00:09.472 ******* 2020-11-09T12:33:30.445 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-11-09T12:33:30.447 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.481 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.042) 0:00:09.515 ******* 2020-11-09T12:33:30.483 INFO:teuthology.task.ansible.out: 2020-11-09T12:33:30.488 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-09T12:33:30.490 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.526 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.044) 0:00:09.560 ******* 2020-11-09T12:33:30.537 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-11-09T12:33:30.539 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi190.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T12:33:30.573 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.047) 0:00:09.607 ******* 2020-11-09T12:33:30.580 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-11-09T12:33:30.581 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.615 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.041) 0:00:09.649 ******* 2020-11-09T12:33:30.622 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-11-09T12:33:30.624 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.661 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.046) 0:00:09.695 ******* 2020-11-09T12:33:30.668 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-09T12:33:30.670 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.703 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.041) 0:00:09.737 ******* 2020-11-09T12:33:30.711 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-11-09T12:33:30.713 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.745 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.041) 0:00:09.779 ******* 2020-11-09T12:33:30.752 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-09T12:33:30.753 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.787 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.041) 0:00:09.821 ******* 2020-11-09T12:33:30.794 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi190.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T12:33:30.829 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.042) 0:00:09.863 ******* 2020-11-09T12:33:30.836 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-11-09T12:33:30.837 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.871 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.042) 0:00:09.905 ******* 2020-11-09T12:33:30.879 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-11-09T12:33:30.881 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.919 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.046) 0:00:09.952 ******* 2020-11-09T12:33:30.925 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:30.962 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:30 +0000 (0:00:00.043) 0:00:09.996 ******* 2020-11-09T12:33:30.971 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi190.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T12:33:31.005 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.042) 0:00:10.039 ******* 2020-11-09T12:33:31.013 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-11-09T12:33:31.015 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.049 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.043) 0:00:10.082 ******* 2020-11-09T12:33:31.056 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-11-09T12:33:31.058 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.093 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.044) 0:00:10.127 ******* 2020-11-09T12:33:31.099 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-11-09T12:33:31.101 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T12:33:31.136 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.042) 0:00:10.170 ******* 2020-11-09T12:33:31.144 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-09T12:33:31.146 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.182 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.046) 0:00:10.216 ******* 2020-11-09T12:33:31.190 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-11-09T12:33:31.191 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.229 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.046) 0:00:10.263 ******* 2020-11-09T12:33:31.236 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-11-09T12:33:31.238 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.277 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.047) 0:00:10.311 ******* 2020-11-09T12:33:31.283 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-11-09T12:33:31.285 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.319 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.041) 0:00:10.352 ******* 2020-11-09T12:33:31.327 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-11-09T12:33:31.329 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.361 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.042) 0:00:10.395 ******* 2020-11-09T12:33:31.367 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-11-09T12:33:31.369 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T12:33:31.402 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.041) 0:00:10.436 ******* 2020-11-09T12:33:31.409 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-11-09T12:33:31.411 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T12:33:31.446 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.043) 0:00:10.480 ******* 2020-11-09T12:33:31.453 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-11-09T12:33:31.455 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.494 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.044) 0:00:10.524 ******* 2020-11-09T12:33:31.497 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-09T12:33:31.499 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.535 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.044) 0:00:10.569 ******* 2020-11-09T12:33:31.542 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-11-09T12:33:31.544 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T12:33:31.579 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.043) 0:00:10.613 ******* 2020-11-09T12:33:31.585 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-11-09T12:33:31.587 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.622 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.043) 0:00:10.656 ******* 2020-11-09T12:33:31.631 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-11-09T12:33:31.633 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.667 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.045) 0:00:10.701 ******* 2020-11-09T12:33:31.674 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-11-09T12:33:31.676 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.709 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.041) 0:00:10.743 ******* 2020-11-09T12:33:31.716 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-09T12:33:31.717 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.750 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.040) 0:00:10.784 ******* 2020-11-09T12:33:31.757 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-11-09T12:33:31.759 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.793 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.043) 0:00:10.827 ******* 2020-11-09T12:33:31.801 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-11-09T12:33:31.802 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.842 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.048) 0:00:10.876 ******* 2020-11-09T12:33:31.849 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-11-09T12:33:31.851 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.888 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.045) 0:00:10.922 ******* 2020-11-09T12:33:31.895 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-11-09T12:33:31.896 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.929 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.041) 0:00:10.963 ******* 2020-11-09T12:33:31.939 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-11-09T12:33:31.940 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:31.973 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:31 +0000 (0:00:00.043) 0:00:11.007 ******* 2020-11-09T12:33:31.980 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-11-09T12:33:31.981 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T12:33:32.017 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.044) 0:00:11.051 ******* 2020-11-09T12:33:32.025 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-09T12:33:32.026 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T12:33:32.060 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.042) 0:00:11.094 ******* 2020-11-09T12:33:32.066 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-09T12:33:32.068 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.102 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.042) 0:00:11.136 ******* 2020-11-09T12:33:32.109 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-11-09T12:33:32.110 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.145 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.043) 0:00:11.179 ******* 2020-11-09T12:33:32.152 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-11-09T12:33:32.154 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.189 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.043) 0:00:11.223 ******* 2020-11-09T12:33:32.197 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-11-09T12:33:32.199 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.257 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.068) 0:00:11.291 ******* 2020-11-09T12:33:32.280 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.304 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.047) 0:00:11.338 ******* 2020-11-09T12:33:32.311 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-09T12:33:32.314 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.348 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.044) 0:00:11.382 ******* 2020-11-09T12:33:32.356 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.392 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.043) 0:00:11.426 ******* 2020-11-09T12:33:32.399 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-11-09T12:33:32.400 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.404 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.436 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.043) 0:00:11.469 ******* 2020-11-09T12:33:32.443 INFO:teuthology.task.ansible.out: 2020-11-09T12:33:32.445 INFO:teuthology.task.ansible.out:TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.486 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.050) 0:00:11.520 ******* 2020-11-09T12:33:32.493 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-11-09T12:33:32.495 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.536 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.050) 0:00:11.570 ******* 2020-11-09T12:33:32.551 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-11-09T12:33:32.553 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.593 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.056) 0:00:11.627 ******* 2020-11-09T12:33:32.604 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-09T12:33:32.606 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.640 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.046) 0:00:11.674 ******* 2020-11-09T12:33:32.646 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-09T12:33:32.648 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T12:33:32.685 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.044) 0:00:11.719 ******* 2020-11-09T12:33:32.712 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-11-09T12:33:32.714 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.750 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.064) 0:00:11.783 ******* 2020-11-09T12:33:32.778 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.812 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.062) 0:00:11.846 ******* 2020-11-09T12:33:32.841 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-09T12:33:32.843 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.877 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.065) 0:00:11.911 ******* 2020-11-09T12:33:32.885 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-09T12:33:32.887 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.091989", "end": "2020-11-09 12:33:32.736734", "rc": 0, "start": "2020-11-09 12:33:32.644745", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-09T12:33:32.921 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.043) 0:00:11.955 ******* 2020-11-09T12:33:32.928 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-11-09T12:33:32.929 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:32.965 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:32 +0000 (0:00:00.043) 0:00:11.999 ******* 2020-11-09T12:33:32.971 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-11-09T12:33:32.973 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.009 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.044) 0:00:12.043 ******* 2020-11-09T12:33:33.016 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-11-09T12:33:33.017 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.053 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.044) 0:00:12.087 ******* 2020-11-09T12:33:33.061 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-09T12:33:33.063 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.097 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.043) 0:00:12.131 ******* 2020-11-09T12:33:33.105 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-09T12:33:33.108 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.140 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.043) 0:00:12.174 ******* 2020-11-09T12:33:33.147 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-09T12:33:33.149 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.182 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.042) 0:00:12.216 ******* 2020-11-09T12:33:33.210 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-11-09T12:33:33.211 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.249 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.066) 0:00:12.283 ******* 2020-11-09T12:33:33.278 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-09T12:33:33.279 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.312 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.063) 0:00:12.346 ******* 2020-11-09T12:33:33.319 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-09T12:33:33.321 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.021404", "end": "2020-11-09 12:33:32.814811", "rc": 0, "start": "2020-11-09 12:33:32.793407", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-09T12:33:33.357 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.044) 0:00:12.391 ******* 2020-11-09T12:33:33.366 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-09T12:33:33.368 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.403 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.045) 0:00:12.437 ******* 2020-11-09T12:33:33.410 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-11-09T12:33:33.412 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.448 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.045) 0:00:12.482 ******* 2020-11-09T12:33:33.456 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.491 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.042) 0:00:12.525 ******* 2020-11-09T12:33:33.498 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-09T12:33:33.499 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.534 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.043) 0:00:12.568 ******* 2020-11-09T12:33:33.562 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-09T12:33:33.564 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.598 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.063) 0:00:12.632 ******* 2020-11-09T12:33:33.628 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-11-09T12:33:33.629 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.011412", "end": "2020-11-09 12:33:33.500466", "rc": 0, "start": "2020-11-09 12:33:33.489054", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:33:33.664 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.065) 0:00:12.698 ******* 2020-11-09T12:33:33.671 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-11-09T12:33:33.672 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.718 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.053) 0:00:12.751 ******* 2020-11-09T12:33:33.725 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-09T12:33:33.727 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.760 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.042) 0:00:12.794 ******* 2020-11-09T12:33:33.767 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-09T12:33:33.769 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.802 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.042) 0:00:12.836 ******* 2020-11-09T12:33:33.830 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-09T12:33:33.832 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.864 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.061) 0:00:12.898 ******* 2020-11-09T12:33:33.891 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-09T12:33:33.893 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:33.925 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.061) 0:00:12.959 ******* 2020-11-09T12:33:33.939 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-09T12:33:33.941 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi190.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.027403", "end": "2020-11-09 12:33:33.795914", "rc": 0, "start": "2020-11-09 12:33:33.768511", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:33:33.974 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:33 +0000 (0:00:00.049) 0:00:13.008 ******* 2020-11-09T12:33:34.985 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.927571", "end": "2020-11-09 12:33:34.500402", "rc": 0, "start": "2020-11-09 12:33:33.572831", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:33:35.019 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:35 +0000 (0:00:01.044) 0:00:14.053 ******* 2020-11-09T12:33:35.046 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-09T12:33:35.048 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:35.080 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:35 +0000 (0:00:00.061) 0:00:14.114 ******* 2020-11-09T12:33:35.224 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-09T12:33:35.225 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.025097", "end": "2020-11-09 12:33:34.745448", "rc": 0, "start": "2020-11-09 12:33:34.720351", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:33:35.258 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:35 +0000 (0:00:00.177) 0:00:14.291 ******* 2020-11-09T12:33:37.207 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-11-09T12:33:37.209 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1604925193, "cache_updated": false, "changed": false} 2020-11-09T12:33:37.248 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:37 +0000 (0:00:01.990) 0:00:16.282 ******* 2020-11-09T12:33:38.130 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-11-09T12:33:38.132 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"cache_update_time": 1604925193, "cache_updated": false, "changed": false} 2020-11-09T12:33:38.165 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:38 +0000 (0:00:00.916) 0:00:17.198 ******* 2020-11-09T12:33:38.172 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-11-09T12:33:38.173 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1604925189, "cache_updated": false, "changed": false} 2020-11-09T12:33:38.207 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:38 +0000 (0:00:00.042) 0:00:17.241 ******* 2020-11-09T12:33:38.214 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-11-09T12:33:38.216 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:38.250 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:38 +0000 (0:00:00.043) 0:00:17.284 ******* 2020-11-09T12:33:38.579 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-11-09T12:33:38.580 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-09T12:33:38.620 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:38 +0000 (0:00:00.369) 0:00:17.654 ******* 2020-11-09T12:33:39.098 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-11-09T12:33:39.099 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"cache_update_time": 1604925189, "cache_updated": false, "changed": false} 2020-11-09T12:33:39.145 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:39 +0000 (0:00:00.513) 0:00:18.167 ******* 2020-11-09T12:33:39.171 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-11-09T12:33:39.173 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:39.205 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:39 +0000 (0:00:00.071) 0:00:18.239 ******* 2020-11-09T12:33:39.487 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-11-09T12:33:39.489 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"cache_update_time": 1604925193, "cache_updated": false, "changed": false} 2020-11-09T12:33:39.539 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:39 +0000 (0:00:00.334) 0:00:18.573 ******* 2020-11-09T12:33:39.547 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-11-09T12:33:39.548 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-09T12:33:39.584 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:39 +0000 (0:00:00.044) 0:00:18.618 ******* 2020-11-09T12:33:39.868 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-09T12:33:39.870 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-09T12:33:40.156 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-09T12:33:40.432 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-09T12:33:40.470 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:40 +0000 (0:00:00.885) 0:00:19.504 ******* 2020-11-09T12:33:40.477 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-11-09T12:33:40.479 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"cache_update_time": 1604925189, "cache_updated": false, "changed": false} 2020-11-09T12:33:40.532 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:40 +0000 (0:00:00.061) 0:00:19.566 ******* 2020-11-09T12:33:40.592 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-11-09T12:33:40.594 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-11-09T12:33:40.646 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:40 +0000 (0:00:00.113) 0:00:19.679 ******* 2020-11-09T12:33:40.890 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-09T12:33:40.892 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-09T12:33:40.966 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-09T12:33:40.968 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-11-09T12:33:41.189 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi164.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-09T12:33:41.231 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-09T12:33:41.233 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-09T12:33:41.467 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-09T12:33:41.469 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-09T12:33:41.504 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:41 +0000 (0:00:00.857) 0:00:20.537 ******* 2020-11-09T12:33:41.511 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-09T12:33:41.513 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-09T12:33:41.550 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:41 +0000 (0:00:00.046) 0:00:20.584 ******* 2020-11-09T12:33:41.590 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-11-09T12:33:41.592 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-11-09T12:33:41.624 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:41 +0000 (0:00:00.074) 0:00:20.658 ******* 2020-11-09T12:33:41.638 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-11-09T12:33:41.641 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-11-09T12:33:41.674 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:41 +0000 (0:00:00.049) 0:00:20.708 ******* 2020-11-09T12:33:41.682 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-11-09T12:33:41.684 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:41.719 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:41 +0000 (0:00:00.044) 0:00:20.753 ******* 2020-11-09T12:33:41.759 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-11-09T12:33:41.762 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T12:33:41.811 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:41 +0000 (0:00:00.091) 0:00:20.845 ******* 2020-11-09T12:33:41.850 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-11-09T12:33:41.852 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:41.863 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:41.874 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:41.876 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:41.912 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:41 +0000 (0:00:00.098) 0:00:20.944 ******* 2020-11-09T12:33:41.937 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:41.971 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:41 +0000 (0:00:00.061) 0:00:21.005 ******* 2020-11-09T12:33:41.988 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi164.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-11-09T12:33:42.262 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-09T12:33:42.558 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-09T12:33:42.597 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:42 +0000 (0:00:00.625) 0:00:21.631 ******* 2020-11-09T12:33:42.636 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-11-09T12:33:42.638 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-11-09T12:33:42.674 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:42 +0000 (0:00:00.076) 0:00:21.708 ******* 2020-11-09T12:33:42.705 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-11-09T12:33:42.707 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:42.750 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:42 +0000 (0:00:00.076) 0:00:21.784 ******* 2020-11-09T12:33:42.779 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-11-09T12:33:42.781 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T12:33:42.826 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:42 +0000 (0:00:00.074) 0:00:21.859 ******* 2020-11-09T12:33:42.854 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-09T12:33:42.856 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"cache_update_time": 1604925193, "cache_updated": false, "changed": false} 2020-11-09T12:33:42.893 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:42 +0000 (0:00:00.067) 0:00:21.927 ******* 2020-11-09T12:33:42.901 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-11-09T12:33:42.902 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:42.903 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:42.939 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:42 +0000 (0:00:00.045) 0:00:21.973 ******* 2020-11-09T12:33:42.981 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-11-09T12:33:42.983 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:43.016 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:43 +0000 (0:00:00.077) 0:00:22.050 ******* 2020-11-09T12:33:43.215 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-11-09T12:33:43.217 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-11-09T12:33:43.253 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:43 +0000 (0:00:00.236) 0:00:22.287 ******* 2020-11-09T12:33:43.579 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-11-09T12:33:43.580 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-11-09T12:33:43.615 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:43 +0000 (0:00:00.361) 0:00:22.649 ******* 2020-11-09T12:33:43.760 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-09T12:33:43.763 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-11-09T12:33:43.868 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-09T12:33:43.906 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:43 +0000 (0:00:00.290) 0:00:22.940 ******* 2020-11-09T12:33:43.924 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-09T12:33:43.926 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"cache_update_time": 1604925189, "cache_updated": false, "changed": false} 2020-11-09T12:33:43.959 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:43 +0000 (0:00:00.053) 0:00:22.993 ******* 2020-11-09T12:33:43.966 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-09T12:33:43.968 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:44.002 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:44 +0000 (0:00:00.042) 0:00:23.036 ******* 2020-11-09T12:33:44.030 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi190.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T12:33:44.066 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:44 +0000 (0:00:00.063) 0:00:23.100 ******* 2020-11-09T12:33:44.104 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-09T12:33:44.106 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:44.142 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:44 +0000 (0:00:00.075) 0:00:23.175 ******* 2020-11-09T12:33:44.278 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-11-09T12:33:44.280 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-11-09T12:33:44.313 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:44 +0000 (0:00:00.171) 0:00:23.347 ******* 2020-11-09T12:33:44.475 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-11-09T12:33:44.477 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-09T12:33:44.520 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:44 +0000 (0:00:00.206) 0:00:23.554 ******* 2020-11-09T12:33:44.641 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-11-09T12:33:44.650 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-11-09T12:33:44.678 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:44 +0000 (0:00:00.157) 0:00:23.711 ******* 2020-11-09T12:33:44.831 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-09T12:33:44.836 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-11-09T12:33:44.936 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-09T12:33:44.972 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:44 +0000 (0:00:00.294) 0:00:24.005 ******* 2020-11-09T12:33:45.001 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-09T12:33:45.003 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:45.036 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.064) 0:00:24.070 ******* 2020-11-09T12:33:45.067 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-09T12:33:45.102 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.066) 0:00:24.136 ******* 2020-11-09T12:33:45.131 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-09T12:33:45.133 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:45.178 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.075) 0:00:24.212 ******* 2020-11-09T12:33:45.199 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-11-09T12:33:45.200 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-11-09 12:32:28 UTC", "ActiveEnterTimestampMonotonic": "17335105", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target nss-lookup.target systemd-journald.socket -.mount system.slice local-fs.target time-sync.target network.target basic.target var-run.mount systemd-tmpfiles-setup.service sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-09 12:32:28 UTC", "AssertTimestampMonotonic": "17334314", "Before": "multi-user.target plymouth-quit.service shutdown.target getty@tty1.service xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-09 12:32:28 UTC", "ConditionTimestampMonotonic": "17334313", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-11-09T12:33:45.202 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1311", "ExecMainStartTimestamp": "Mon 2020-11-09 12:32:28 UTC", "ExecMainStartTimestampMonotonic": "17335079", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-09 12:32:28 UTC", "InactiveExitTimestampMonotonic": "17335105", "InvocationID": "ac13f86e9e544cf487610956bfe1f77d", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2020-11-09T12:33:45.203 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1311", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2020-11-09T12:33:45.205 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-09 12:32:28 UTC", "StateChangeTimestampMonotonic": "17335105", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-09 12:32:28 UTC", "WatchdogTimestampMonotonic": "17335104", "WatchdogUSec": "0"}} 2020-11-09T12:33:45.237 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.059) 0:00:24.271 ******* 2020-11-09T12:33:45.266 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-11-09T12:33:45.267 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:45.306 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.068) 0:00:24.340 ******* 2020-11-09T12:33:45.336 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:45.383 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.076) 0:00:24.417 ******* 2020-11-09T12:33:45.429 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:45.460 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.076) 0:00:24.494 ******* 2020-11-09T12:33:45.490 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:45.533 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.070) 0:00:24.565 ******* 2020-11-09T12:33:45.538 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-09T12:33:45.590 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.052) 0:00:24.617 ******* 2020-11-09T12:33:45.598 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:45.629 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.044) 0:00:24.662 ******* 2020-11-09T12:33:45.682 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:45.723 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.095) 0:00:24.757 ******* 2020-11-09T12:33:45.753 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-09T12:33:45.755 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:45.789 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.065) 0:00:24.822 ******* 2020-11-09T12:33:45.806 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-11-09T12:33:45.807 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-11-09 12:32:34 UTC", "ActiveEnterTimestampMonotonic": "17871434", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service systemd-journald.socket basic.target local-fs.target var-run.mount -.mount system.slice sysinit.target network.target nss-lookup.target time-sync.target remote-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-09 12:32:34 UTC", "AssertTimestampMonotonic": "17870266", "Before": "plymouth-quit.service multi-user.target getty@tty1.service xdm.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-09 12:32:34 UTC", "ConditionTimestampMonotonic": "17870266", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-11-09T12:33:45.808 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1431", "ExecMainStartTimestamp": "Mon 2020-11-09 12:32:34 UTC", "ExecMainStartTimestampMonotonic": "17871405", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-09 12:32:34 UTC", "InactiveExitTimestampMonotonic": "17871434", "InvocationID": "88e7b198f88640f2bfc30d9b3bca8dfd", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2020-11-09T12:33:45.810 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1431", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2020-11-09T12:33:45.811 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-09 12:32:34 UTC", "StateChangeTimestampMonotonic": "17871434", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-09 12:32:34 UTC", "WatchdogTimestampMonotonic": "17871433", "WatchdogUSec": "0"}} 2020-11-09T12:33:45.844 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.055) 0:00:24.878 ******* 2020-11-09T12:33:45.852 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-11-09T12:33:45.854 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:45.892 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.047) 0:00:24.926 ******* 2020-11-09T12:33:45.899 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:45.937 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.044) 0:00:24.971 ******* 2020-11-09T12:33:45.943 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-11-09T12:33:45.945 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:45.979 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:45 +0000 (0:00:00.042) 0:00:25.013 ******* 2020-11-09T12:33:45.986 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:46.027 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.048) 0:00:25.061 ******* 2020-11-09T12:33:46.034 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-11-09T12:33:46.036 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:46.073 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.045) 0:00:25.107 ******* 2020-11-09T12:33:46.081 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-11-09T12:33:46.083 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:46.124 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.051) 0:00:25.158 ******* 2020-11-09T12:33:46.131 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-11-09T12:33:46.133 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-09T12:33:46.167 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.042) 0:00:25.201 ******* 2020-11-09T12:33:46.173 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-11-09T12:33:46.174 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:46.223 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.054) 0:00:25.256 ******* 2020-11-09T12:33:46.229 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-09T12:33:46.229 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-09T12:33:46.275 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.053) 0:00:25.309 ******* 2020-11-09T12:33:46.282 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-11-09T12:33:46.284 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:46.322 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.047) 0:00:25.356 ******* 2020-11-09T12:33:46.329 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-09T12:33:46.330 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-09T12:33:46.366 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.043) 0:00:25.400 ******* 2020-11-09T12:33:46.374 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-11-09T12:33:46.376 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:46.422 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.055) 0:00:25.456 ******* 2020-11-09T12:33:46.430 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-11-09T12:33:46.432 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-09T12:33:46.469 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.046) 0:00:25.503 ******* 2020-11-09T12:33:46.475 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-09T12:33:46.477 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:46.513 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.044) 0:00:25.547 ******* 2020-11-09T12:33:46.542 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-11-09T12:33:46.544 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:46.578 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.063) 0:00:25.611 ******* 2020-11-09T12:33:46.598 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-09T12:33:46.599 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-09T12:33:46.634 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.056) 0:00:25.668 ******* 2020-11-09T12:33:46.641 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-11-09T12:33:46.642 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:46.695 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.060) 0:00:25.728 ******* 2020-11-09T12:33:46.724 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:46.759 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.064) 0:00:25.793 ******* 2020-11-09T12:33:46.778 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-11-09T12:33:46.779 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "gid": 1234, "name": "ubuntu", "state": "present", "system": false} 2020-11-09T12:33:46.816 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.057) 0:00:25.850 ******* 2020-11-09T12:33:46.824 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-09T12:33:46.860 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.043) 0:00:25.894 ******* 2020-11-09T12:33:46.867 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:46.906 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.046) 0:00:25.940 ******* 2020-11-09T12:33:46.914 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* ok: [smithi164.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-09T12:33:46.956 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.049) 0:00:25.990 ******* 2020-11-09T12:33:46.964 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-09T12:33:46.966 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:47.000 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:46 +0000 (0:00:00.043) 0:00:26.034 ******* 2020-11-09T12:33:47.007 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-09T12:33:47.009 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-09T12:33:47.048 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:47 +0000 (0:00:00.047) 0:00:26.081 ******* 2020-11-09T12:33:47.090 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-11-09T12:33:47.092 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-09T12:33:47.125 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:47 +0000 (0:00:00.077) 0:00:26.159 ******* 2020-11-09T12:33:47.157 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-09T12:33:47.158 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-09T12:33:47.194 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:47 +0000 (0:00:00.068) 0:00:26.228 ******* 2020-11-09T12:33:47.199 INFO:teuthology.task.ansible.out: 2020-11-09T12:33:47.255 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-09T12:33:47.298 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:47 +0000 (0:00:00.103) 0:00:26.332 ******* 2020-11-09T12:33:47.432 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-11-09T12:33:47.434 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 1234, "name": "ubuntu", "state": "present", "system": false} 2020-11-09T12:33:47.466 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:47 +0000 (0:00:00.168) 0:00:26.500 ******* 2020-11-09T12:33:47.493 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-11-09T12:33:47.495 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:47.529 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:47 +0000 (0:00:00.062) 0:00:26.563 ******* 2020-11-09T12:33:47.557 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-09T12:33:47.559 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:47.594 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:47 +0000 (0:00:00.062) 0:00:26.626 ******* 2020-11-09T12:33:47.599 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-09T12:33:47.645 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:47 +0000 (0:00:00.052) 0:00:26.678 ******* 2020-11-09T12:33:47.685 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-09T12:33:47.687 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi190"}, "changed": false} 2020-11-09T12:33:47.729 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:47 +0000 (0:00:00.081) 0:00:26.760 ******* 2020-11-09T12:33:47.749 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-09T12:33:47.750 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-09T12:33:47.803 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:47 +0000 (0:00:00.077) 0:00:26.837 ******* 2020-11-09T12:33:48.190 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-09T12:33:48.192 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-09T12:33:48.224 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:48 +0000 (0:00:00.420) 0:00:27.258 ******* 2020-11-09T12:33:48.251 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-09T12:33:48.253 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi164"}, "changed": false} 2020-11-09T12:33:48.285 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:48 +0000 (0:00:00.061) 0:00:27.319 ******* 2020-11-09T12:33:48.621 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ ok: [smithi190.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi190", "ansible_hostname": "smithi190", "ansible_nodename": "smithi190"}, "changed": false, "name": "smithi190"} 2020-11-09T12:33:48.655 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:48 +0000 (0:00:00.369) 0:00:27.689 ******* 2020-11-09T12:33:48.757 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi164", "ansible_hostname": "smithi164", "ansible_nodename": "smithi164"}, "changed": false, "name": "smithi164"} 2020-11-09T12:33:48.795 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:48 +0000 (0:00:00.139) 0:00:27.829 ******* 2020-11-09T12:33:48.995 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-09T12:33:49.030 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:49 +0000 (0:00:00.235) 0:00:28.064 ******* 2020-11-09T12:33:49.122 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-09T12:33:49.159 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:49 +0000 (0:00:00.127) 0:00:28.192 ******* 2020-11-09T12:33:49.356 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-11-09T12:33:49.358 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-09T12:33:49.396 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:49 +0000 (0:00:00.238) 0:00:28.430 ******* 2020-11-09T12:33:49.468 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-09T12:33:49.507 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:49 +0000 (0:00:00.109) 0:00:28.540 ******* 2020-11-09T12:33:49.919 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-11-09T12:33:49.921 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-09T12:33:49.955 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:49 +0000 (0:00:00.448) 0:00:28.988 ******* 2020-11-09T12:33:49.993 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.037 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.083) 0:00:29.071 ******* 2020-11-09T12:33:50.189 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-11-09T12:33:50.191 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-09T12:33:50.227 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.189) 0:00:29.261 ******* 2020-11-09T12:33:50.235 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-09T12:33:50.237 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.281 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.054) 0:00:29.315 ******* 2020-11-09T12:33:50.291 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-11-09T12:33:50.293 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.329 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.047) 0:00:29.363 ******* 2020-11-09T12:33:50.336 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-09T12:33:50.338 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.383 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.051) 0:00:29.414 ******* 2020-11-09T12:33:50.390 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-09T12:33:50.392 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.432 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.050) 0:00:29.465 ******* 2020-11-09T12:33:50.443 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-09T12:33:50.445 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.486 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.050) 0:00:29.516 ******* 2020-11-09T12:33:50.489 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-09T12:33:50.491 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.524 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.042) 0:00:29.558 ******* 2020-11-09T12:33:50.532 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-09T12:33:50.534 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.567 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.042) 0:00:29.601 ******* 2020-11-09T12:33:50.574 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-09T12:33:50.576 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.613 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.046) 0:00:29.647 ******* 2020-11-09T12:33:50.621 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-09T12:33:50.623 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.658 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.044) 0:00:29.692 ******* 2020-11-09T12:33:50.665 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-09T12:33:50.667 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.704 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.046) 0:00:29.738 ******* 2020-11-09T12:33:50.715 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.749 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.044) 0:00:29.783 ******* 2020-11-09T12:33:50.756 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-09T12:33:50.758 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.793 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.043) 0:00:29.827 ******* 2020-11-09T12:33:50.801 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.845 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.052) 0:00:29.879 ******* 2020-11-09T12:33:50.852 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.888 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.042) 0:00:29.922 ******* 2020-11-09T12:33:50.896 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-11-09T12:33:50.935 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.047) 0:00:29.969 ******* 2020-11-09T12:33:50.942 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-11-09T12:33:50.944 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:50.985 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:50 +0000 (0:00:00.049) 0:00:30.019 ******* 2020-11-09T12:33:50.993 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-09T12:33:51.027 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.042) 0:00:30.061 ******* 2020-11-09T12:33:51.036 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-11-09T12:33:51.071 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.043) 0:00:30.105 ******* 2020-11-09T12:33:51.078 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-09T12:33:51.113 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.042) 0:00:30.147 ******* 2020-11-09T12:33:51.121 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-09T12:33:51.155 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.042) 0:00:30.189 ******* 2020-11-09T12:33:51.163 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.200 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.043) 0:00:30.233 ******* 2020-11-09T12:33:51.208 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-09T12:33:51.247 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.047) 0:00:30.281 ******* 2020-11-09T12:33:51.256 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-11-09T12:33:51.258 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.305 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.057) 0:00:30.339 ******* 2020-11-09T12:33:51.312 INFO:teuthology.task.ansible.out: 2020-11-09T12:33:51.314 INFO:teuthology.task.ansible.out:TASK [testnode : Install GPG] ************************************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.349 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.044) 0:00:30.383 ******* 2020-11-09T12:33:51.356 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-11-09T12:33:51.358 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.394 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.044) 0:00:30.427 ******* 2020-11-09T12:33:51.401 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi164.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.404 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.445 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.051) 0:00:30.479 ******* 2020-11-09T12:33:51.454 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:33:51.456 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.494 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.048) 0:00:30.527 ******* 2020-11-09T12:33:51.500 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-11-09T12:33:51.502 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.538 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.044) 0:00:30.572 ******* 2020-11-09T12:33:51.546 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-11-09T12:33:51.548 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.584 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.045) 0:00:30.617 ******* 2020-11-09T12:33:51.589 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:33:51.591 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.628 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.045) 0:00:30.662 ******* 2020-11-09T12:33:51.635 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-11-09T12:33:51.637 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.673 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.044) 0:00:30.707 ******* 2020-11-09T12:33:51.679 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-11-09T12:33:51.681 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.715 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.042) 0:00:30.749 ******* 2020-11-09T12:33:51.722 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-11-09T12:33:51.723 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.757 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.041) 0:00:30.791 ******* 2020-11-09T12:33:51.763 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-11-09T12:33:51.765 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.798 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.040) 0:00:30.832 ******* 2020-11-09T12:33:51.805 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-11-09T12:33:51.806 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.840 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.042) 0:00:30.874 ******* 2020-11-09T12:33:51.850 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-11-09T12:33:51.852 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.891 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.051) 0:00:30.925 ******* 2020-11-09T12:33:51.899 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-09T12:33:51.901 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.938 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.046) 0:00:30.971 ******* 2020-11-09T12:33:51.946 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-11-09T12:33:51.948 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:51.985 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:51 +0000 (0:00:00.047) 0:00:31.019 ******* 2020-11-09T12:33:51.992 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-09T12:33:51.994 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.028 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.042) 0:00:31.062 ******* 2020-11-09T12:33:52.035 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-09T12:33:52.036 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.084 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.055) 0:00:31.118 ******* 2020-11-09T12:33:52.092 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-09T12:33:52.094 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.134 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.049) 0:00:31.167 ******* 2020-11-09T12:33:52.140 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-09T12:33:52.142 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.175 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.041) 0:00:31.209 ******* 2020-11-09T12:33:52.183 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-11-09T12:33:52.185 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.218 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.043) 0:00:31.252 ******* 2020-11-09T12:33:52.225 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-09T12:33:52.227 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.263 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.045) 0:00:31.297 ******* 2020-11-09T12:33:52.271 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-09T12:33:52.272 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.312 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.048) 0:00:31.346 ******* 2020-11-09T12:33:52.319 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-11-09T12:33:52.321 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.355 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.042) 0:00:31.389 ******* 2020-11-09T12:33:52.362 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-09T12:33:52.364 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.398 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.042) 0:00:31.432 ******* 2020-11-09T12:33:52.404 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-09T12:33:52.406 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.444 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.046) 0:00:31.478 ******* 2020-11-09T12:33:52.451 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-11-09T12:33:52.453 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.487 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.043) 0:00:31.521 ******* 2020-11-09T12:33:52.495 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-09T12:33:52.497 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.531 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.044) 0:00:31.565 ******* 2020-11-09T12:33:52.540 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-11-09T12:33:52.541 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.579 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.045) 0:00:31.611 ******* 2020-11-09T12:33:52.584 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-11-09T12:33:52.586 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.619 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.042) 0:00:31.653 ******* 2020-11-09T12:33:52.626 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-11-09T12:33:52.627 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.663 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.043) 0:00:31.697 ******* 2020-11-09T12:33:52.670 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-11-09T12:33:52.672 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.720 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.056) 0:00:31.754 ******* 2020-11-09T12:33:52.728 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.766 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.046) 0:00:31.800 ******* 2020-11-09T12:33:52.773 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-11-09T12:33:52.774 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.809 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.043) 0:00:31.843 ******* 2020-11-09T12:33:52.819 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-11-09T12:33:52.820 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.853 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.043) 0:00:31.887 ******* 2020-11-09T12:33:52.860 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-11-09T12:33:52.861 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.896 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.042) 0:00:31.930 ******* 2020-11-09T12:33:52.903 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-11-09T12:33:52.904 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.939 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.042) 0:00:31.972 ******* 2020-11-09T12:33:52.947 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:52.985 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:52 +0000 (0:00:00.046) 0:00:32.019 ******* 2020-11-09T12:33:52.992 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-11-09T12:33:52.994 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:53.027 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:53 +0000 (0:00:00.041) 0:00:32.061 ******* 2020-11-09T12:33:53.033 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-11-09T12:33:53.035 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:53.070 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:53 +0000 (0:00:00.042) 0:00:32.103 ******* 2020-11-09T12:33:53.077 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-11-09T12:33:53.079 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:53.112 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:53 +0000 (0:00:00.043) 0:00:32.146 ******* 2020-11-09T12:33:53.121 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-11-09T12:33:53.123 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:53.169 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:53 +0000 (0:00:00.056) 0:00:32.203 ******* 2020-11-09T12:33:53.198 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-11-09T12:33:53.200 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:53.244 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:53 +0000 (0:00:00.075) 0:00:32.278 ******* 2020-11-09T12:33:53.272 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-11-09T12:33:53.274 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005224", "end": "2020-11-09 12:33:53.150336", "rc": 0, "start": "2020-11-09 12:33:53.145112", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:33:53.310 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:53 +0000 (0:00:00.065) 0:00:32.343 ******* 2020-11-09T12:33:53.339 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-11-09T12:33:53.387 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:53 +0000 (0:00:00.077) 0:00:32.421 ******* 2020-11-09T12:33:53.429 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-11-09T12:33:53.431 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005285", "end": "2020-11-09 12:33:52.954022", "rc": 0, "start": "2020-11-09 12:33:52.948737", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:33:53.466 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:53 +0000 (0:00:00.078) 0:00:32.500 ******* 2020-11-09T12:33:53.494 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-11-09T12:33:53.533 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:53 +0000 (0:00:00.067) 0:00:32.567 ******* 2020-11-09T12:33:53.743 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-11-09T12:33:53.745 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-11-09T12:33:53.779 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:53 +0000 (0:00:00.245) 0:00:32.813 ******* 2020-11-09T12:33:53.808 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-11-09T12:33:53.810 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:53.842 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:53 +0000 (0:00:00.063) 0:00:32.876 ******* 2020-11-09T12:33:53.881 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-11-09T12:33:53.885 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-11-09T12:33:53.917 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:53 +0000 (0:00:00.075) 0:00:32.951 ******* 2020-11-09T12:33:53.946 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-11-09T12:33:53.948 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:33:53.982 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:53 +0000 (0:00:00.064) 0:00:33.016 ******* 2020-11-09T12:33:55.164 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-11-09T12:33:55.166 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-09T12:33:55.186 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-09T12:33:56.119 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-09T12:33:56.120 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-09T12:33:56.154 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:56 +0000 (0:00:02.172) 0:00:35.188 ******* 2020-11-09T12:33:56.189 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:56 +0000 (0:00:00.034) 0:00:35.222 ******* 2020-11-09T12:33:57.055 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-11-09T12:33:57.057 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"cache_update_time": 1604925189, "cache_updated": false, "changed": false} 2020-11-09T12:33:57.091 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:57 +0000 (0:00:00.902) 0:00:36.125 ******* 2020-11-09T12:33:57.099 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"cache_update_time": 1604925193, "cache_updated": false, "changed": false} 2020-11-09T12:33:57.143 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:57 +0000 (0:00:00.052) 0:00:36.177 ******* 2020-11-09T12:33:57.151 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-09T12:33:57.190 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:57 +0000 (0:00:00.047) 0:00:36.224 ******* 2020-11-09T12:33:57.237 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:57 +0000 (0:00:00.046) 0:00:36.271 ******* 2020-11-09T12:33:57.377 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-11-09T12:33:57.379 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.035796", "end": "2020-11-09 12:33:56.896850", "rc": 0, "start": "2020-11-09 12:33:56.861054", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:33:57.415 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:57 +0000 (0:00:00.177) 0:00:36.448 ******* 2020-11-09T12:33:57.424 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.035677", "end": "2020-11-09 12:33:57.291030", "rc": 0, "start": "2020-11-09 12:33:57.255353", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:33:57.460 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:33:57 +0000 (0:00:00.045) 0:00:36.494 ******* 2020-11-09T12:34:00.291 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-11-09T12:34:00.293 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"cache_update_time": 1604925189, "cache_updated": false, "changed": false} 2020-11-09T12:34:00.327 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:00 +0000 (0:00:02.866) 0:00:39.361 ******* 2020-11-09T12:34:00.335 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"cache_update_time": 1604925193, "cache_updated": false, "changed": false} 2020-11-09T12:34:00.370 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:00 +0000 (0:00:00.043) 0:00:39.404 ******* 2020-11-09T12:34:10.303 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-11-09T12:34:10.305 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102070 files and directories 2020-11-09T12:34:10.306 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-11-09T12:34:10.307 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102070 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-09T12:34:10.342 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:10 +0000 (0:00:09.971) 0:00:49.376 ******* 2020-11-09T12:34:10.579 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102070 files and directories 2020-11-09T12:34:10.581 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-11-09T12:34:10.582 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102070 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-09T12:34:10.615 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:10 +0000 (0:00:00.272) 0:00:49.649 ******* 2020-11-09T12:34:11.249 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-09T12:34:11.251 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"cache_update_time": 1604925193, "cache_updated": false, "changed": false} 2020-11-09T12:34:11.283 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:11 +0000 (0:00:00.668) 0:00:50.317 ******* 2020-11-09T12:34:11.310 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-11-09T12:34:11.312 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:11.343 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:11 +0000 (0:00:00.060) 0:00:50.377 ******* 2020-11-09T12:34:11.496 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-09T12:34:11.498 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"cache_update_time": 1604925189, "cache_updated": false, "changed": false} 2020-11-09T12:34:11.533 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:11 +0000 (0:00:00.189) 0:00:50.567 ******* 2020-11-09T12:34:11.560 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:11.594 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:11 +0000 (0:00:00.060) 0:00:50.628 ******* 2020-11-09T12:34:19.839 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-09T12:34:19.841 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"cache_update_time": 1604925193, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]\nFetched 10.4 MB in 1s (9939 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-11-09T12:34:19.842 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101928 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-11-09T12:34:19.844 INFO:teuthology.task.ansible.out:1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]", "Fetched 10.4 MB in 1s (9939 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-11-09T12:34:19.845 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101928 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-09T12:34:19.874 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:19 +0000 (0:00:08.280) 0:00:58.908 ******* 2020-11-09T12:34:20.266 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"cache_update_time": 1604925189, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]\nFetched 10.4 MB in 1s (9313 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-11-09T12:34:20.268 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101928 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-11-09T12:34:20.269 INFO:teuthology.task.ansible.out:1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]", "Fetched 10.4 MB in 1s (9313 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-11-09T12:34:20.271 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101928 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-09T12:34:20.303 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:20 +0000 (0:00:00.428) 0:00:59.337 ******* 2020-11-09T12:34:20.740 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-11-09T12:34:20.742 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"cache_update_time": 1604925193, "cache_updated": false, "changed": false} 2020-11-09T12:34:20.776 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:20 +0000 (0:00:00.473) 0:00:59.810 ******* 2020-11-09T12:34:20.897 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-09T12:34:20.931 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:20 +0000 (0:00:00.155) 0:00:59.965 ******* 2020-11-09T12:34:20.959 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-09T12:34:20.961 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:20.992 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:20 +0000 (0:00:00.060) 0:01:00.026 ******* 2020-11-09T12:34:21.122 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-09T12:34:21.124 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-11-09T12:34:21.155 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:21 +0000 (0:00:00.163) 0:01:00.189 ******* 2020-11-09T12:34:21.161 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-11-09T12:34:21.163 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"cache_update_time": 1604925189, "cache_updated": false, "changed": false} 2020-11-09T12:34:21.197 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:21 +0000 (0:00:00.041) 0:01:00.231 ******* 2020-11-09T12:34:21.318 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-11-09T12:34:21.320 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-09T12:34:21.352 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:21 +0000 (0:00:00.155) 0:01:00.386 ******* 2020-11-09T12:34:21.379 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:21.412 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:21 +0000 (0:00:00.060) 0:01:00.446 ******* 2020-11-09T12:34:21.450 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-09T12:34:21.452 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-11-09T12:34:21.485 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:21 +0000 (0:00:00.072) 0:01:00.519 ******* 2020-11-09T12:34:21.534 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-09T12:34:21.536 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-11-09T12:34:21.570 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:21 +0000 (0:00:00.085) 0:01:00.604 ******* 2020-11-09T12:34:21.641 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-09T12:34:21.643 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1234, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-09T12:34:21.676 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:21 +0000 (0:00:00.105) 0:01:00.710 ******* 2020-11-09T12:34:21.807 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-11-09T12:34:21.809 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T12:34:21.841 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:21 +0000 (0:00:00.165) 0:01:00.875 ******* 2020-11-09T12:34:21.881 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-09T12:34:21.882 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-11-09T12:34:21.918 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:21 +0000 (0:00:00.075) 0:01:00.951 ******* 2020-11-09T12:34:22.071 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-09T12:34:22.072 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1234, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-09T12:34:22.109 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.192) 0:01:01.143 ******* 2020-11-09T12:34:22.127 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-09T12:34:22.129 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Mon 2020-11-09 12:32:28 UTC", "ActiveEnterTimestampMonotonic": "17781191", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target basic.target remote-fs.target -.mount network.target systemd-tmpfiles-setup.service system.slice nss-lookup.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-09 12:32:28 UTC", "AssertTimestampMonotonic": "17335968", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-09 12:32:28 UTC", "ConditionTimestampMonotonic": "17335968", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-11-09T12:34:22.130 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1408", "ExecMainStartTimestamp": "Mon 2020-11-09 12:32:28 UTC", "ExecMainStartTimestampMonotonic": "17781179", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-09 12:32:28 UTC", "InactiveExitTimestampMonotonic": "17336970", "InvocationID": "c5e2ad9f1ceb48debcfe272ea853e727", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-11-09T12:34:22.131 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1408", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-11-09T12:34:22.133 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-09 12:32:28 UTC", "StateChangeTimestampMonotonic": "17781191", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-09 12:32:28 UTC", "WatchdogTimestampMonotonic": "17781189", "WatchdogUSec": "0"}} 2020-11-09T12:34:22.164 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.054) 0:01:01.198 ******* 2020-11-09T12:34:22.192 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-09T12:34:22.193 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.225 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.060) 0:01:01.259 ******* 2020-11-09T12:34:22.246 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-11-09T12:34:22.248 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.280 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.055) 0:01:01.314 ******* 2020-11-09T12:34:22.311 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.031) 0:01:01.345 ******* 2020-11-09T12:34:22.339 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-09T12:34:22.341 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.373 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.061) 0:01:01.407 ******* 2020-11-09T12:34:22.399 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-09T12:34:22.401 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.434 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.061) 0:01:01.468 ******* 2020-11-09T12:34:22.462 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-09T12:34:22.464 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.498 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.063) 0:01:01.532 ******* 2020-11-09T12:34:22.525 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-09T12:34:22.527 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.558 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.060) 0:01:01.592 ******* 2020-11-09T12:34:22.567 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-09T12:34:22.568 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Mon 2020-11-09 12:32:35 UTC", "ActiveEnterTimestampMonotonic": "18317952", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount nss-lookup.target system.slice sysinit.target systemd-journald.socket basic.target remote-fs.target network.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-09 12:32:34 UTC", "AssertTimestampMonotonic": "17867960", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-09 12:32:34 UTC", "ConditionTimestampMonotonic": "17867960", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-11-09T12:34:22.570 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1543", "ExecMainStartTimestamp": "Mon 2020-11-09 12:32:35 UTC", "ExecMainStartTimestampMonotonic": "18317938", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-09 12:32:34 UTC", "InactiveExitTimestampMonotonic": "17868611", "InvocationID": "5a35d5efab864c20a31f794a990b8af8", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-11-09T12:34:22.571 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1543", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-11-09T12:34:22.572 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-09 12:32:35 UTC", "StateChangeTimestampMonotonic": "18317952", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-09 12:32:35 UTC", "WatchdogTimestampMonotonic": "18317951", "WatchdogUSec": "0"}} 2020-11-09T12:34:22.603 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.045) 0:01:01.637 ******* 2020-11-09T12:34:22.612 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-09T12:34:22.613 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.645 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.041) 0:01:01.679 ******* 2020-11-09T12:34:22.651 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-09T12:34:22.653 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.686 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.041) 0:01:01.720 ******* 2020-11-09T12:34:22.693 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-09T12:34:22.695 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.727 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.041) 0:01:01.761 ******* 2020-11-09T12:34:22.733 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-09T12:34:22.735 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.771 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.044) 0:01:01.805 ******* 2020-11-09T12:34:22.778 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:34:22.779 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.812 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.041) 0:01:01.846 ******* 2020-11-09T12:34:22.818 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-09T12:34:22.820 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.853 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.040) 0:01:01.887 ******* 2020-11-09T12:34:22.860 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-09T12:34:22.862 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.894 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.041) 0:01:01.928 ******* 2020-11-09T12:34:22.901 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-09T12:34:22.902 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.935 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.041) 0:01:01.969 ******* 2020-11-09T12:34:22.945 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-09T12:34:22.947 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:22.979 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:22 +0000 (0:00:00.043) 0:01:02.013 ******* 2020-11-09T12:34:22.986 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-09T12:34:22.987 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.021 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.041) 0:01:02.055 ******* 2020-11-09T12:34:23.027 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-09T12:34:23.029 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.061 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.040) 0:01:02.095 ******* 2020-11-09T12:34:23.068 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-09T12:34:23.070 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.104 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.043) 0:01:02.138 ******* 2020-11-09T12:34:23.112 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-09T12:34:23.114 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.147 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.042) 0:01:02.181 ******* 2020-11-09T12:34:23.154 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-09T12:34:23.155 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.189 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.041) 0:01:02.223 ******* 2020-11-09T12:34:23.196 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-09T12:34:23.197 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.232 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.043) 0:01:02.266 ******* 2020-11-09T12:34:23.240 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-09T12:34:23.241 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.276 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.043) 0:01:02.310 ******* 2020-11-09T12:34:23.283 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-09T12:34:23.284 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.318 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.041) 0:01:02.352 ******* 2020-11-09T12:34:23.324 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:34:23.325 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.362 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.044) 0:01:02.396 ******* 2020-11-09T12:34:23.368 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-09T12:34:23.370 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.402 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.040) 0:01:02.436 ******* 2020-11-09T12:34:23.409 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-09T12:34:23.411 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.443 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.041) 0:01:02.477 ******* 2020-11-09T12:34:23.450 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-09T12:34:23.451 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.484 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.041) 0:01:02.518 ******* 2020-11-09T12:34:23.490 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-09T12:34:23.492 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.525 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.040) 0:01:02.559 ******* 2020-11-09T12:34:23.532 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-09T12:34:23.534 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.566 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.040) 0:01:02.600 ******* 2020-11-09T12:34:23.572 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-09T12:34:23.574 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.607 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.040) 0:01:02.641 ******* 2020-11-09T12:34:23.614 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-09T12:34:23.616 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.648 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.040) 0:01:02.682 ******* 2020-11-09T12:34:23.654 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-09T12:34:23.655 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.688 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.040) 0:01:02.722 ******* 2020-11-09T12:34:23.696 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-09T12:34:23.698 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.730 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.041) 0:01:02.764 ******* 2020-11-09T12:34:23.737 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-09T12:34:23.738 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.772 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.041) 0:01:02.806 ******* 2020-11-09T12:34:23.779 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-09T12:34:23.780 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.813 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.041) 0:01:02.847 ******* 2020-11-09T12:34:23.820 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-09T12:34:23.821 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.854 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.041) 0:01:02.888 ******* 2020-11-09T12:34:23.861 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-09T12:34:23.863 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.895 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.041) 0:01:02.929 ******* 2020-11-09T12:34:23.901 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-09T12:34:23.903 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.940 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.044) 0:01:02.974 ******* 2020-11-09T12:34:23.947 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-09T12:34:23.949 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:23.981 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:23 +0000 (0:00:00.041) 0:01:03.015 ******* 2020-11-09T12:34:23.989 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-09T12:34:23.990 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:24.022 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.040) 0:01:03.056 ******* 2020-11-09T12:34:24.029 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-09T12:34:24.030 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:24.064 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.042) 0:01:03.098 ******* 2020-11-09T12:34:24.071 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:24.106 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.041) 0:01:03.140 ******* 2020-11-09T12:34:24.113 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-09T12:34:24.114 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:24.148 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.042) 0:01:03.182 ******* 2020-11-09T12:34:24.155 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:24.193 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.044) 0:01:03.227 ******* 2020-11-09T12:34:24.201 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-11-09T12:34:24.203 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:24.245 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.051) 0:01:03.279 ******* 2020-11-09T12:34:24.253 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:24.301 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.055) 0:01:03.334 ******* 2020-11-09T12:34:24.330 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-09T12:34:24.332 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:24.376 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.075) 0:01:03.410 ******* 2020-11-09T12:34:24.396 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-11-09T12:34:24.397 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-11-09T12:34:24.429 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.052) 0:01:03.463 ******* 2020-11-09T12:34:24.436 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-09T12:34:24.437 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:24.471 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.042) 0:01:03.505 ******* 2020-11-09T12:34:24.500 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-09T12:34:24.502 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:24.534 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.062) 0:01:03.568 ******* 2020-11-09T12:34:24.562 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-09T12:34:24.564 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:24.595 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.060) 0:01:03.629 ******* 2020-11-09T12:34:24.622 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-09T12:34:24.624 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:24.660 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.065) 0:01:03.694 ******* 2020-11-09T12:34:24.688 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:24.722 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.061) 0:01:03.756 ******* 2020-11-09T12:34:24.824 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-09T12:34:24.826 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-11-09T12:34:24.860 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.137) 0:01:03.894 ******* 2020-11-09T12:34:24.867 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-11-09T12:34:24.869 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-11-09T12:34:24.903 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.042) 0:01:03.937 ******* 2020-11-09T12:34:24.911 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-09T12:34:24.913 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:24.946 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:24 +0000 (0:00:00.043) 0:01:03.980 ******* 2020-11-09T12:34:25.229 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-09T12:34:25.231 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-11-09T12:34:25.266 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:25 +0000 (0:00:00.319) 0:01:04.300 ******* 2020-11-09T12:34:25.295 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-09T12:34:25.297 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:25.331 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:25 +0000 (0:00:00.065) 0:01:04.365 ******* 2020-11-09T12:34:25.340 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-11-09T12:34:25.342 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-11-09T12:34:25.384 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:25 +0000 (0:00:00.052) 0:01:04.418 ******* 2020-11-09T12:34:25.652 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-11-09T12:34:25.688 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:25 +0000 (0:00:00.303) 0:01:04.722 ******* 2020-11-09T12:34:25.717 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-11-09T12:34:25.752 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:25 +0000 (0:00:00.063) 0:01:04.786 ******* 2020-11-09T12:34:26.019 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-11-09T12:34:26.055 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.303) 0:01:05.089 ******* 2020-11-09T12:34:26.072 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-09T12:34:26.074 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-11-09T12:34:26.107 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.052) 0:01:05.141 ******* 2020-11-09T12:34:26.244 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-09T12:34:26.245 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T12:34:26.276 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.169) 0:01:05.310 ******* 2020-11-09T12:34:26.303 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-09T12:34:26.305 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.336 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.059) 0:01:05.370 ******* 2020-11-09T12:34:26.364 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-09T12:34:26.365 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-11-09T12:34:26.398 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.061) 0:01:05.432 ******* 2020-11-09T12:34:26.430 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.032) 0:01:05.464 ******* 2020-11-09T12:34:26.457 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-09T12:34:26.459 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.491 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.060) 0:01:05.525 ******* 2020-11-09T12:34:26.518 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-09T12:34:26.520 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-09T12:34:26.521 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T12:34:26.553 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.062) 0:01:05.587 ******* 2020-11-09T12:34:26.586 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.032) 0:01:05.619 ******* 2020-11-09T12:34:26.592 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-09T12:34:26.594 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.629 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.043) 0:01:05.663 ******* 2020-11-09T12:34:26.636 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-09T12:34:26.637 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.670 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.040) 0:01:05.704 ******* 2020-11-09T12:34:26.676 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-09T12:34:26.677 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.712 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.042) 0:01:05.746 ******* 2020-11-09T12:34:26.719 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:34:26.721 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-09T12:34:26.754 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.041) 0:01:05.788 ******* 2020-11-09T12:34:26.762 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-09T12:34:26.763 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.796 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.042) 0:01:05.830 ******* 2020-11-09T12:34:26.805 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-09T12:34:26.806 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.816 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.818 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.828 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-09T12:34:26.830 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi190.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.863 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.066) 0:01:05.897 ******* 2020-11-09T12:34:26.870 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-09T12:34:26.871 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.872 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.872 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.883 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.885 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.894 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.896 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-09T12:34:26.897 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:26.932 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.069) 0:01:05.966 ******* 2020-11-09T12:34:26.969 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:26 +0000 (0:00:00.036) 0:01:06.003 ******* 2020-11-09T12:34:26.983 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:34:26.984 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-09T12:34:27.019 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:27 +0000 (0:00:00.049) 0:01:06.052 ******* 2020-11-09T12:34:27.067 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-09T12:34:27.069 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.079 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.090 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.101 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.103 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.113 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-09T12:34:27.124 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.135 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.137 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.147 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.148 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.158 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.170 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.208 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:27 +0000 (0:00:00.189) 0:01:06.242 ******* 2020-11-09T12:34:27.215 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-09T12:34:27.217 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003659", "end": "2020-11-09 12:34:27.172158", "rc": 0, "start": "2020-11-09 12:34:27.168499", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-09T12:34:27.254 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:27 +0000 (0:00:00.045) 0:01:06.288 ******* 2020-11-09T12:34:27.281 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-09T12:34:27.283 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.318 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:27 +0000 (0:00:00.064) 0:01:06.352 ******* 2020-11-09T12:34:27.356 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-11-09T12:34:27.357 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.390 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:27 +0000 (0:00:00.071) 0:01:06.424 ******* 2020-11-09T12:34:27.397 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-09T12:34:27.398 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003372", "end": "2020-11-09 12:34:27.370340", "rc": 0, "start": "2020-11-09 12:34:27.366968", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-09T12:34:27.435 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:27 +0000 (0:00:00.045) 0:01:06.469 ******* 2020-11-09T12:34:27.442 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.476 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:27 +0000 (0:00:00.040) 0:01:06.510 ******* 2020-11-09T12:34:27.482 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-09T12:34:27.484 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.521 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:27 +0000 (0:00:00.044) 0:01:06.555 ******* 2020-11-09T12:34:27.559 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.598 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:27 +0000 (0:00:00.076) 0:01:06.632 ******* 2020-11-09T12:34:27.625 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-09T12:34:27.627 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:27.658 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:27 +0000 (0:00:00.060) 0:01:06.692 ******* 2020-11-09T12:34:28.361 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-11-09T12:34:28.363 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"cache_update_time": 1604925193, "cache_updated": false, "changed": false} 2020-11-09T12:34:28.398 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:28 +0000 (0:00:00.737) 0:01:07.430 ******* 2020-11-09T12:34:28.423 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-11-09T12:34:28.425 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:28.459 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:28 +0000 (0:00:00.062) 0:01:07.493 ******* 2020-11-09T12:34:28.496 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-09T12:34:28.497 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item={'block_used': 13640640, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226454468, 'size_available': 927557500928, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113945, 'block_size': 4096, 'inode_available': 60941031}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226454468, "block_size": 4096, "block_total": 240095108, "block_used": 13640640, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941031, "inode_total": 61054976, "inode_used": 113945, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927557500928, "size_total": 983429562368, "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}} 2020-11-09T12:34:28.531 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:28 +0000 (0:00:00.072) 0:01:07.565 ******* 2020-11-09T12:34:28.552 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-11-09T12:34:28.553 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"cache_update_time": 1604925189, "cache_updated": false, "changed": false} 2020-11-09T12:34:28.588 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:28 +0000 (0:00:00.056) 0:01:07.622 ******* 2020-11-09T12:34:28.615 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:28.649 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:28 +0000 (0:00:00.061) 0:01:07.683 ******* 2020-11-09T12:34:28.666 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-11-09T12:34:28.668 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009947", "end": "2020-11-09 12:34:28.573181", "rc": 0, "start": "2020-11-09 12:34:28.563234", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:28.701 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:28 +0000 (0:00:00.051) 0:01:07.735 ******* 2020-11-09T12:34:28.708 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-09T12:34:28.709 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'block_used': 13640639, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226454469, 'size_available': 927557505024, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113945, 'block_size': 4096, 'inode_available': 60941031}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226454469, "block_size": 4096, "block_total": 240095108, "block_used": 13640639, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941031, "inode_total": 61054976, "inode_used": 113945, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927557505024, "size_total": 983429562368, "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}} 2020-11-09T12:34:28.744 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:28 +0000 (0:00:00.043) 0:01:07.778 ******* 2020-11-09T12:34:28.752 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-09T12:34:28.754 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'block_used': 13640640, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226454468, 'size_available': 927557500928, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113945, 'block_size': 4096, 'inode_available': 60941031}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226454468, "block_size": 4096, "block_total": 240095108, "block_used": 13640640, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941031, "inode_total": 61054976, "inode_used": 113945, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927557500928, "size_total": 983429562368, "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:28.788 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:28 +0000 (0:00:00.043) 0:01:07.822 ******* 2020-11-09T12:34:28.815 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-09T12:34:28.850 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:28 +0000 (0:00:00.061) 0:01:07.884 ******* 2020-11-09T12:34:28.881 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-11-09T12:34:28.883 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009680", "end": "2020-11-09 12:34:28.423304", "rc": 0, "start": "2020-11-09 12:34:28.413624", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:28.915 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:28 +0000 (0:00:00.065) 0:01:07.949 ******* 2020-11-09T12:34:28.922 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-09T12:34:28.924 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:28.955 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-09T12:34:28.956 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'block_used': 13640639, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226454469, 'size_available': 927557505024, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113945, 'block_size': 4096, 'inode_available': 60941031}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226454469, "block_size": 4096, "block_total": 240095108, "block_used": 13640639, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941031, "inode_total": 61054976, "inode_used": 113945, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927557505024, "size_total": 983429562368, "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:28.997 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:28 +0000 (0:00:00.075) 0:01:08.025 ******* 2020-11-09T12:34:29.022 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-09T12:34:29.068 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:29 +0000 (0:00:00.076) 0:01:08.102 ******* 2020-11-09T12:34:29.115 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-09T12:34:29.117 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:31.169 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkwzeDTYUkbEw1XvVUnZ8gJfOapVyZZ8IXT'], 'uuids': ['c86d298a-4d40-4bd2-b859-4e989f45615d']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.147454", "end": "2020-11-09 12:34:31.058956", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkwzeDTYUkbEw1XvVUnZ8gJfOapVyZZ8IXT"], "labels": [], "masters": [], "uuids": ["c86d298a-4d40-4bd2-b859-4e989f45615d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:28.911502", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T12:34:31.171 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:31.192 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:31.235 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8Yku0mrfyDu61AnSghubBavEAUnCgX8KPnW'], 'uuids': ['d8d35401-ee6b-4da4-92ad-0b47561bb23a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.992940", "end": "2020-11-09 12:34:30.768648", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8Yku0mrfyDu61AnSghubBavEAUnCgX8KPnW"], "labels": [], "masters": [], "uuids": ["d8d35401-ee6b-4da4-92ad-0b47561bb23a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:28.775708", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T12:34:31.247 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:31.248 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:32.432 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkwtT1kCKPyhoIvkqHWzQz5vz72Yp0RrPO6'], 'uuids': ['922b9c70-6c55-446d-be4c-951c73046e14']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.126891", "end": "2020-11-09 12:34:32.334854", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkwtT1kCKPyhoIvkqHWzQz5vz72Yp0RrPO6"], "labels": [], "masters": [], "uuids": ["922b9c70-6c55-446d-be4c-951c73046e14"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:31.207963", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T12:34:32.444 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKCA', 'wwn-0x5000c50092937ba8'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKCA-part1', 'wwn-0x5000c50092937ba8-part1'], 'uuids': ['1e9a17af-0168-46be-b05d-6a296317ab15']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092937ba8', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XKCA', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKCA", "wwn-0x5000c50092937ba8"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKCA-part1", "wwn-0x5000c50092937ba8-part1"], "labels": [], "masters": [], "uuids": ["1e9a17af-0168-46be-b05d-6a296317ab15"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XKCA", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092937ba8"}}, "skip_reason": 2020-11-09T12:34:32.446 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-09T12:34:32.508 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8YkjUVdGBYWlQa9znrd28M3T1MMYv5J6z9W'], 'uuids': ['4a1e9615-91d5-4f92-8544-4e4eee223f67']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.138230", "end": "2020-11-09 12:34:32.052270", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8YkjUVdGBYWlQa9znrd28M3T1MMYv5J6z9W"], "labels": [], "masters": [], "uuids": ["4a1e9615-91d5-4f92-8544-4e4eee223f67"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:30.914040", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T12:34:32.520 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTQ7', 'wwn-0x5000c5009293f07d'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTQ7-part1', 'wwn-0x5000c5009293f07d-part1'], 'uuids': ['1e9a17af-0168-46be-b05d-6a296317ab15']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009293f07d', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XTQ7', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTQ7", "wwn-0x5000c5009293f07d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTQ7-part1", "wwn-0x5000c5009293f07d-part1"], "labels": [], "masters": [], "uuids": ["1e9a17af-0168-46be-b05d-6a296317ab15"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XTQ7", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293f07d"}}, "skip_reason": 2020-11-09T12:34:32.522 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-09T12:34:33.736 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-JkeryM-iLVS-Wg1H-GdAr-5Gl6-CpUu-95b0eW', 'nvme-INTEL_SSDPEDMD400G4_CVFT63210012400BGN', 'nvme-nvme.8086-43564654363332313030313234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.180830", "end": "2020-11-09 12:34:33.641001", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-JkeryM-iLVS-Wg1H-GdAr-5Gl6-CpUu-95b0eW", "nvme-INTEL_SSDPEDMD400G4_CVFT63210012400BGN", "nvme-nvme.8086-43564654363332313030313234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:32.460171", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-09T12:34:33.738 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T12:34:33.748 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:33.760 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:33.792 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-ryK44i-i9CG-6XuS-LH0s-CHMk-MtVr-6HiqQt', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300H0400BGN', 'nvme-nvme.8086-43564654363233333030483034303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.151910", "end": "2020-11-09 12:34:33.333787", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ryK44i-i9CG-6XuS-LH0s-CHMk-MtVr-6HiqQt", "nvme-INTEL_SSDPEDMD400G4_CVFT623300H0400BGN", "nvme-nvme.8086-43564654363233333030483034303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:32.181877", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-09T12:34:33.794 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T12:34:33.804 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:33.826 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:35.012 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkwUjsuIBNiMfaBnM7x1dI5GHO2MJLEsmJn'], 'uuids': ['3d2dcdb9-b67b-47c1-8965-aedeb28531d8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.132515", "end": "2020-11-09 12:34:34.905034", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkwUjsuIBNiMfaBnM7x1dI5GHO2MJLEsmJn"], "labels": [], "masters": [], "uuids": ["3d2dcdb9-b67b-47c1-8965-aedeb28531d8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:33.772519", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T12:34:35.024 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:35.024 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:35.088 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8YkmubBLKRfFWRVLuRYnkCojBs23CTL8pAh'], 'uuids': ['66104e1d-ac46-45c4-b76e-fa1edbaf64db']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.141676", "end": "2020-11-09 12:34:34.631704", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8YkmubBLKRfFWRVLuRYnkCojBs23CTL8pAh"], "labels": [], "masters": [], "uuids": ["66104e1d-ac46-45c4-b76e-fa1edbaf64db"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:33.490028", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T12:34:35.099 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:35.111 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:36.262 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkw2lH58kDNe2jcgYjEky1W1GlaBiHXh2Vd'], 'uuids': ['d82b16d3-e477-44f1-a4b1-21d52240fcdc']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.113946", "end": "2020-11-09 12:34:36.166478", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkw2lH58kDNe2jcgYjEky1W1GlaBiHXh2Vd"], "labels": [], "masters": [], "uuids": ["d82b16d3-e477-44f1-a4b1-21d52240fcdc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:35.052532", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T12:34:36.274 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:36.360 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8YktN93o4Oqvyf1PpLMNt9VhJMDcmWQw9i1'], 'uuids': ['ea5043e3-225f-4640-a174-32dde6da810d']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.121367", "end": "2020-11-09 12:34:35.901948", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8YktN93o4Oqvyf1PpLMNt9VhJMDcmWQw9i1"], "labels": [], "masters": [], "uuids": ["ea5043e3-225f-4640-a174-32dde6da810d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:34.780581", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T12:34:36.372 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:37.631 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8Yk0vg83OpYaXY38IHDVSLU5YLb6Gqdiem7'], 'uuids': ['ea58feb2-d084-4fd9-8915-78325122c797']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.138515", "end": "2020-11-09 12:34:37.170893", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8Yk0vg83OpYaXY38IHDVSLU5YLb6Gqdiem7"], "labels": [], "masters": [], "uuids": ["ea58feb2-d084-4fd9-8915-78325122c797"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:36.032378", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T12:34:37.674 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:37 +0000 (0:00:08.606) 0:01:16.708 ******* 2020-11-09T12:34:37.723 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-09T12:34:37.724 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:37.756 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-09T12:34:37.758 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkw84sJvBL8QReUlTuvMw2X84pnWYsQAHGb'], 'uuids': ['f2597693-c4d5-4936-9161-5c8956f4db34']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.359254", "end": "2020-11-09 12:34:37.650107", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkw84sJvBL8QReUlTuvMw2X84pnWYsQAHGb"], "labels": [], "masters": [], "uuids": ["f2597693-c4d5-4936-9161-5c8956f4db34"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:36.290853", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-09T12:34:37.806 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:37 +0000 (0:00:00.131) 0:01:16.839 ******* 2020-11-09T12:34:37.856 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-09T12:34:37.858 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:38.002 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8Yku0mrfyDu61AnSghubBavEAUnCgX8KPnW'], 'uuids': ['d8d35401-ee6b-4da4-92ad-0b47561bb23a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.158697", "end": "2020-11-09 12:34:37.548318", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8Yku0mrfyDu61AnSghubBavEAUnCgX8KPnW"], "labels": [], "masters": [], "uuids": ["d8d35401-ee6b-4da4-92ad-0b47561bb23a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:37.389621", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153947 s, 749 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153947 s, 749 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:38.003 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:38.015 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:38.130 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkwzeDTYUkbEw1XvVUnZ8gJfOapVyZZ8IXT'], 'uuids': ['c86d298a-4d40-4bd2-b859-4e989f45615d']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.156549", "end": "2020-11-09 12:34:38.032590", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkwzeDTYUkbEw1XvVUnZ8gJfOapVyZZ8IXT"], "labels": [], "masters": [], "uuids": ["c86d298a-4d40-4bd2-b859-4e989f45615d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:37.876041", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151839 s, 760 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151839 s, 760 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:38.141 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:38.153 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:38.300 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8YkjUVdGBYWlQa9znrd28M3T1MMYv5J6z9W'], 'uuids': ['4a1e9615-91d5-4f92-8544-4e4eee223f67']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.165485", "end": "2020-11-09 12:34:37.846874", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8YkjUVdGBYWlQa9znrd28M3T1MMYv5J6z9W"], "labels": [], "masters": [], "uuids": ["4a1e9615-91d5-4f92-8544-4e4eee223f67"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:37.681389", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.160755 s, 718 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.160755 s, 718 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:38.302 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTQ7', 'wwn-0x5000c5009293f07d'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTQ7-part1', 'wwn-0x5000c5009293f07d-part1'], 'uuids': ['1e9a17af-0168-46be-b05d-6a296317ab15']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009293f07d', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XTQ7', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTQ7", "wwn-0x5000c5009293f07d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTQ7-part1", "wwn-0x5000c5009293f07d-part1"], "labels": [], "masters": [], "uuids": ["1e9a17af-0168-46be-b05d-6a296317ab15"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XTQ7", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293f07d"}}, "skip_reason": 2020-11-09T12:34:38.303 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-09T12:34:38.427 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkwtT1kCKPyhoIvkqHWzQz5vz72Yp0RrPO6'], 'uuids': ['922b9c70-6c55-446d-be4c-951c73046e14']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.162774", "end": "2020-11-09 12:34:38.337481", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkwtT1kCKPyhoIvkqHWzQz5vz72Yp0RrPO6"], "labels": [], "masters": [], "uuids": ["922b9c70-6c55-446d-be4c-951c73046e14"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:38.174707", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.159753 s, 722 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.159753 s, 722 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:38.439 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKCA', 'wwn-0x5000c50092937ba8'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKCA-part1', 'wwn-0x5000c50092937ba8-part1'], 'uuids': ['1e9a17af-0168-46be-b05d-6a296317ab15']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092937ba8', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XKCA', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKCA", "wwn-0x5000c50092937ba8"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKCA-part1", "wwn-0x5000c50092937ba8-part1"], "labels": [], "masters": [], "uuids": ["1e9a17af-0168-46be-b05d-6a296317ab15"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XKCA", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092937ba8"}}, "skip_reason": 2020-11-09T12:34:38.441 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-09T12:34:38.482 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-ryK44i-i9CG-6XuS-LH0s-CHMk-MtVr-6HiqQt', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300H0400BGN', 'nvme-nvme.8086-43564654363233333030483034303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.052910", "end": "2020-11-09 12:34:38.025198", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ryK44i-i9CG-6XuS-LH0s-CHMk-MtVr-6HiqQt", "nvme-INTEL_SSDPEDMD400G4_CVFT623300H0400BGN", "nvme-nvme.8086-43564654363233333030483034303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:37.972288", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0499029 s, 2.3 GB/s", "std 2020-11-09T12:34:38.484 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0499029 s, 2.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:38.494 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:38.496 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:38.611 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-JkeryM-iLVS-Wg1H-GdAr-5Gl6-CpUu-95b0eW', 'nvme-INTEL_SSDPEDMD400G4_CVFT63210012400BGN', 'nvme-nvme.8086-43564654363332313030313234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.051310", "end": "2020-11-09 12:34:38.512339", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-JkeryM-iLVS-Wg1H-GdAr-5Gl6-CpUu-95b0eW", "nvme-INTEL_SSDPEDMD400G4_CVFT63210012400BGN", "nvme-nvme.8086-43564654363332313030313234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:38.461029", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0466121 s, 2.5 GB/s", "std 2020-11-09T12:34:38.613 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0466121 s, 2.5 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:38.622 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:38.624 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:38.780 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8YkmubBLKRfFWRVLuRYnkCojBs23CTL8pAh'], 'uuids': ['66104e1d-ac46-45c4-b76e-fa1edbaf64db']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.157456", "end": "2020-11-09 12:34:38.327559", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8YkmubBLKRfFWRVLuRYnkCojBs23CTL8pAh"], "labels": [], "masters": [], "uuids": ["66104e1d-ac46-45c4-b76e-fa1edbaf64db"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:38.170103", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154012 s, 749 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154012 s, 749 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:38.791 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:38.803 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:38.887 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkwUjsuIBNiMfaBnM7x1dI5GHO2MJLEsmJn'], 'uuids': ['3d2dcdb9-b67b-47c1-8965-aedeb28531d8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.148786", "end": "2020-11-09 12:34:38.793981", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkwUjsuIBNiMfaBnM7x1dI5GHO2MJLEsmJn"], "labels": [], "masters": [], "uuids": ["3d2dcdb9-b67b-47c1-8965-aedeb28531d8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:38.645195", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144403 s, 799 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144403 s, 799 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:38.898 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:38.910 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:39.090 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8YktN93o4Oqvyf1PpLMNt9VhJMDcmWQw9i1'], 'uuids': ['ea5043e3-225f-4640-a174-32dde6da810d']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.159600", "end": "2020-11-09 12:34:38.629759", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8YktN93o4Oqvyf1PpLMNt9VhJMDcmWQw9i1"], "labels": [], "masters": [], "uuids": ["ea5043e3-225f-4640-a174-32dde6da810d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:38.470159", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154891 s, 745 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154891 s, 745 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:39.091 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:39.186 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkw2lH58kDNe2jcgYjEky1W1GlaBiHXh2Vd'], 'uuids': ['d82b16d3-e477-44f1-a4b1-21d52240fcdc']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.156112", "end": "2020-11-09 12:34:39.083020", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkw2lH58kDNe2jcgYjEky1W1GlaBiHXh2Vd"], "labels": [], "masters": [], "uuids": ["d82b16d3-e477-44f1-a4b1-21d52240fcdc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:38.926908", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151385 s, 762 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151385 s, 762 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:39.198 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:39.375 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8Yk0vg83OpYaXY38IHDVSLU5YLb6Gqdiem7'], 'uuids': ['ea58feb2-d084-4fd9-8915-78325122c797']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.156739", "end": "2020-11-09 12:34:38.916072", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-HpY8xkQZHH1Mo38qk34xw3BrzxhDI8Yk0vg83OpYaXY38IHDVSLU5YLb6Gqdiem7"], "labels": [], "masters": [], "uuids": ["ea58feb2-d084-4fd9-8915-78325122c797"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:38.759333", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152008 s, 759 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152008 s, 759 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:39.437 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:39 +0000 (0:00:01.631) 0:01:18.471 ******* 2020-11-09T12:34:39.478 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkw84sJvBL8QReUlTuvMw2X84pnWYsQAHGb'], 'uuids': ['f2597693-c4d5-4936-9161-5c8956f4db34']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.156530", "end": "2020-11-09 12:34:39.372093", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-Xee1EszyfJoBfGSUbqRqLUNU7FwNykkw84sJvBL8QReUlTuvMw2X84pnWYsQAHGb"], "labels": [], "masters": [], "uuids": ["f2597693-c4d5-4936-9161-5c8956f4db34"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-09 12:34:39.215563", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151811 s, 760 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151811 s, 760 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:39.523 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:39 +0000 (0:00:00.085) 0:01:18.557 ******* 2020-11-09T12:34:39.749 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-11-09T12:34:39.751 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.160941", "end": "2020-11-09 12:34:39.289622", "rc": 0, "start": "2020-11-09 12:34:39.128681", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:39.788 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:39 +0000 (0:00:00.264) 0:01:18.822 ******* 2020-11-09T12:34:39.809 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.134381", "end": "2020-11-09 12:34:39.692927", "rc": 0, "start": "2020-11-09 12:34:39.558546", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:39.846 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:39 +0000 (0:00:00.057) 0:01:18.880 ******* 2020-11-09T12:34:40.030 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-11-09T12:34:40.032 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.048292", "end": "2020-11-09 12:34:39.932396", "rc": 0, "start": "2020-11-09 12:34:39.884104", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:40.064 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.217) 0:01:19.098 ******* 2020-11-09T12:34:40.091 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-09T12:34:40.126 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.062) 0:01:19.160 ******* 2020-11-09T12:34:40.154 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-09T12:34:40.156 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:40.190 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.063) 0:01:19.223 ******* 2020-11-09T12:34:40.217 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-11-09T12:34:40.218 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.288910", "end": "2020-11-09 12:34:39.761692", "rc": 0, "start": "2020-11-09 12:34:39.472782", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:40.250 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.060) 0:01:19.284 ******* 2020-11-09T12:34:40.281 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.031) 0:01:19.315 ******* 2020-11-09T12:34:40.287 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-09T12:34:40.326 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.045) 0:01:19.360 ******* 2020-11-09T12:34:40.333 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-09T12:34:40.334 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:40.369 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.042) 0:01:19.403 ******* 2020-11-09T12:34:40.375 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-09T12:34:40.377 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:40.411 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.042) 0:01:19.445 ******* 2020-11-09T12:34:40.417 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-09T12:34:40.419 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:40.452 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.040) 0:01:19.486 ******* 2020-11-09T12:34:40.458 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-11-09T12:34:40.460 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:40.495 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.043) 0:01:19.529 ******* 2020-11-09T12:34:40.523 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-09T12:34:40.525 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:40.559 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.064) 0:01:19.593 ******* 2020-11-09T12:34:40.578 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-11-09T12:34:40.580 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T12:34:40.613 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.053) 0:01:19.647 ******* 2020-11-09T12:34:40.620 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-09T12:34:40.622 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:40.655 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.042) 0:01:19.689 ******* 2020-11-09T12:34:40.663 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-09T12:34:40.664 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'block_used': 13640640, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226454468, 'size_available': 927557500928, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113945, 'block_size': 4096, 'inode_available': 60941031}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226454468, "block_size": 4096, "block_total": 240095108, "block_used": 13640640, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941031, "inode_total": 61054976, "inode_used": 113945, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927557500928, "size_total": 983429562368, "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:40.700 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.044) 0:01:19.734 ******* 2020-11-09T12:34:40.727 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:40.761 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.060) 0:01:19.795 ******* 2020-11-09T12:34:40.777 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-11-09T12:34:40.779 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-09T12:34:40.811 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.050) 0:01:19.845 ******* 2020-11-09T12:34:40.819 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:34:40.820 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:40.854 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.042) 0:01:19.888 ******* 2020-11-09T12:34:40.861 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-09T12:34:40.862 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'block_used': 13640639, 'uuid': '1e9a17af-0168-46be-b05d-6a296317ab15', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226454469, 'size_available': 927557505024, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113945, 'block_size': 4096, 'inode_available': 60941031}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226454469, "block_size": 4096, "block_total": 240095108, "block_used": 13640639, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941031, "inode_total": 61054976, "inode_used": 113945, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927557505024, "size_total": 983429562368, "uuid": "1e9a17af-0168-46be-b05d-6a296317ab15"}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:40.898 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.044) 0:01:19.932 ******* 2020-11-09T12:34:40.914 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:34:40.916 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:40.948 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.049) 0:01:19.982 ******* 2020-11-09T12:34:40.955 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-11-09T12:34:40.956 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:40.990 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:40 +0000 (0:00:00.042) 0:01:20.024 ******* 2020-11-09T12:34:40.997 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-09T12:34:40.999 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:41.032 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:41 +0000 (0:00:00.041) 0:01:20.066 ******* 2020-11-09T12:34:41.039 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:34:41.041 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:41.074 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:41 +0000 (0:00:00.041) 0:01:20.108 ******* 2020-11-09T12:34:41.081 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-11-09T12:34:41.083 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:41.118 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:41 +0000 (0:00:00.044) 0:01:20.152 ******* 2020-11-09T12:34:41.125 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:34:41.126 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:41.160 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:41 +0000 (0:00:00.041) 0:01:20.194 ******* 2020-11-09T12:34:41.167 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-09T12:34:41.168 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:41.201 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:41 +0000 (0:00:00.041) 0:01:20.235 ******* 2020-11-09T12:34:41.209 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-09T12:34:41.211 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:41.243 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:41 +0000 (0:00:00.042) 0:01:20.277 ******* 2020-11-09T12:34:41.283 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-11-09T12:34:41.284 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:41.316 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:41 +0000 (0:00:00.072) 0:01:20.350 ******* 2020-11-09T12:34:41.343 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-09T12:34:41.345 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:41.377 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:41 +0000 (0:00:00.060) 0:01:20.411 ******* 2020-11-09T12:34:41.789 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-11-09T12:34:41.791 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-09T12:34:41.824 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:41 +0000 (0:00:00.447) 0:01:20.858 ******* 2020-11-09T12:34:41.856 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:41 +0000 (0:00:00.031) 0:01:20.889 ******* 2020-11-09T12:34:42.384 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-09T12:34:42.386 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T12:34:42.396 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T12:34:42.626 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T12:34:42.637 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T12:34:42.836 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T12:34:42.878 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T12:34:43.065 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T12:34:43.111 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T12:34:43.298 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T12:34:43.334 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:43 +0000 (0:00:01.478) 0:01:22.368 ******* 2020-11-09T12:34:43.393 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-09T12:34:43.430 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:43 +0000 (0:00:00.096) 0:01:22.464 ******* 2020-11-09T12:34:43.459 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-09T12:34:43.461 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-09T12:34:43.494 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:43 +0000 (0:00:00.063) 0:01:22.528 ******* 2020-11-09T12:34:43.564 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-09T12:34:43.599 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:43 +0000 (0:00:00.104) 0:01:22.633 ******* 2020-11-09T12:34:43.649 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-11-09T12:34:43.651 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T12:34:43.745 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T12:34:43.756 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T12:34:43.862 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T12:34:43.864 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T12:34:43.978 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T12:34:43.979 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T12:34:43.992 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:44.028 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:44 +0000 (0:00:00.429) 0:01:23.062 ******* 2020-11-09T12:34:44.087 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-09T12:34:44.098 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:44.135 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:44 +0000 (0:00:00.107) 0:01:23.169 ******* 2020-11-09T12:34:44.153 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-11-09T12:34:44.155 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-11-09T12:34:44.188 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:44 +0000 (0:00:00.052) 0:01:23.222 ******* 2020-11-09T12:34:44.226 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-11-09T12:34:44.227 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-11-09T12:34:44.266 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:44 +0000 (0:00:00.078) 0:01:23.300 ******* 2020-11-09T12:34:44.274 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-11-09T12:34:44.275 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-11-09T12:34:44.310 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:44 +0000 (0:00:00.043) 0:01:23.344 ******* 2020-11-09T12:34:44.349 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-11-09T12:34:44.351 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-11-09T12:34:44.382 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:44 +0000 (0:00:00.071) 0:01:23.416 ******* 2020-11-09T12:34:44.755 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-11-09T12:34:44.757 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true} 2020-11-09T12:34:44.790 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:44 +0000 (0:00:00.408) 0:01:23.824 ******* 2020-11-09T12:34:45.242 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-11-09T12:34:45.244 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-09T12:34:45.275 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:45 +0000 (0:00:00.485) 0:01:24.309 ******* 2020-11-09T12:34:45.417 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-11-09T12:34:45.419 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 1234, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-11-09T12:34:45.450 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:45 +0000 (0:00:00.175) 0:01:24.484 ******* 2020-11-09T12:34:45.591 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-11-09T12:34:45.593 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-09T12:34:45.626 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:45 +0000 (0:00:00.175) 0:01:24.659 ******* 2020-11-09T12:34:45.758 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-11-09T12:34:45.760 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"changed": true} 2020-11-09T12:34:45.793 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:45 +0000 (0:00:00.167) 0:01:24.827 ******* 2020-11-09T12:34:45.976 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-11-09T12:34:45.978 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1234, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi190.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-09T12:34:46.011 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:46 +0000 (0:00:00.217) 0:01:25.045 ******* 2020-11-09T12:34:46.044 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:46 +0000 (0:00:00.033) 0:01:25.078 ******* 2020-11-09T12:34:46.188 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-11-09T12:34:46.189 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "gid": 1234, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-11-09T12:34:46.222 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:46 +0000 (0:00:00.177) 0:01:25.256 ******* 2020-11-09T12:34:46.332 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-09T12:34:46.333 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-09T12:34:46.368 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:46 +0000 (0:00:00.145) 0:01:25.402 ******* 2020-11-09T12:34:46.375 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-11-09T12:34:46.377 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-09T12:34:46.411 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:46 +0000 (0:00:00.043) 0:01:25.445 ******* 2020-11-09T12:34:46.760 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-11-09T12:34:46.761 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1234, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-11-09T12:34:46.795 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:46 +0000 (0:00:00.383) 0:01:25.829 ******* 2020-11-09T12:34:47.111 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-09T12:34:47.113 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-09T12:34:47.152 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:47 +0000 (0:00:00.357) 0:01:26.186 ******* 2020-11-09T12:34:47.263 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-09T12:34:47.265 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"cache_update_time": 1604925189, "cache_updated": false, "changed": false} 2020-11-09T12:34:47.298 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:47 +0000 (0:00:00.145) 0:01:26.332 ******* 2020-11-09T12:34:47.484 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-09T12:34:47.485 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047978", "end": "2020-11-09 12:34:47.036733", "rc": 0, "start": "2020-11-09 12:34:46.988755", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-09T12:34:47.518 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:47 +0000 (0:00:00.219) 0:01:26.552 ******* 2020-11-09T12:34:47.560 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:47.594 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:47 +0000 (0:00:00.075) 0:01:26.628 ******* 2020-11-09T12:34:47.620 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:47.654 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:47 +0000 (0:00:00.059) 0:01:26.687 ******* 2020-11-09T12:34:48.060 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-09T12:34:48.061 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"cache_update_time": 1604925193, "cache_updated": false, "changed": false} 2020-11-09T12:34:48.093 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:48 +0000 (0:00:00.439) 0:01:27.127 ******* 2020-11-09T12:34:48.265 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-09T12:34:48.267 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.048184", "end": "2020-11-09 12:34:48.179356", "rc": 0, "start": "2020-11-09 12:34:48.131172", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-09T12:34:48.299 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:48 +0000 (0:00:00.206) 0:01:27.333 ******* 2020-11-09T12:34:48.327 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-11-09T12:34:48.329 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:48.360 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:48 +0000 (0:00:00.060) 0:01:27.394 ******* 2020-11-09T12:34:48.387 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-11-09T12:34:48.389 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:48.421 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:48 +0000 (0:00:00.060) 0:01:27.455 ******* 2020-11-09T12:34:48.541 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-09T12:34:48.543 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"cache_update_time": 1604925189, "cache_updated": false, "changed": false} 2020-11-09T12:34:48.577 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:48 +0000 (0:00:00.156) 0:01:27.611 ******* 2020-11-09T12:34:48.604 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-09T12:34:48.606 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:48.638 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:48 +0000 (0:00:00.060) 0:01:27.672 ******* 2020-11-09T12:34:48.958 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-09T12:34:48.960 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-11-09T12:34:49.003 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:49 +0000 (0:00:00.365) 0:01:28.037 ******* 2020-11-09T12:34:49.032 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-11-09T12:34:49.034 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:49.065 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:49 +0000 (0:00:00.061) 0:01:28.099 ******* 2020-11-09T12:34:49.271 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-09T12:34:49.271 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-11-09 12:32:40 UTC", "ActiveEnterTimestampMonotonic": "24024300", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice basic.target -.mount systemd-journald.socket systemd-tmpfiles-setup.service sysinit.target network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-09 12:32:34 UTC", "AssertTimestampMonotonic": "17584158", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-09 12:32:34 UTC", "ConditionTimestampMonotonic": "17584157", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-11-09T12:34:49.273 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1968", "ExecMainStartTimestamp": "Mon 2020-11-09 12:32:40 UTC", "ExecMainStartTimestampMonotonic": "24024287", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-09 12:32:34 UTC", "InactiveExitTimestampMonotonic": "17584949", "InvocationID": "7d622433ee3e4f8fb7661c2ddb6bf3c3", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-11-09T12:34:49.274 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1968", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-09 12:32:40 UTC", "S 2020-11-09T12:34:49.276 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "24024300", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-09 12:32:40 UTC", "WatchdogTimestampMonotonic": "24024299", "WatchdogUSec": "0"}} 2020-11-09T12:34:49.311 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:49 +0000 (0:00:00.241) 0:01:28.341 ******* 2020-11-09T12:34:49.314 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-09T12:34:49.316 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"cache_update_time": 1604925193, "cache_updated": false, "changed": false} 2020-11-09T12:34:49.354 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:49 +0000 (0:00:00.047) 0:01:28.388 ******* 2020-11-09T12:34:49.361 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:34:49.362 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-09T12:34:49.396 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:49 +0000 (0:00:00.041) 0:01:28.430 ******* 2020-11-09T12:34:49.404 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-09T12:34:49.406 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:49.439 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:49 +0000 (0:00:00.043) 0:01:28.473 ******* 2020-11-09T12:34:49.446 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:34:49.448 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:49.482 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:49 +0000 (0:00:00.043) 0:01:28.516 ******* 2020-11-09T12:34:49.510 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-09T12:34:49.512 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:49.543 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:49 +0000 (0:00:00.061) 0:01:28.577 ******* 2020-11-09T12:34:49.758 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-09T12:34:49.760 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-11-09T12:34:49.793 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:49 +0000 (0:00:00.249) 0:01:28.827 ******* 2020-11-09T12:34:49.820 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:49.854 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:49 +0000 (0:00:00.060) 0:01:28.888 ******* 2020-11-09T12:34:50.059 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-09T12:34:50.061 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-11-09 12:32:34 UTC", "ActiveEnterTimestampMonotonic": "23339221", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket system.slice network.target -.mount basic.target systemd-tmpfiles-setup.service sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-11-09 12:32:27 UTC", "AssertTimestampMonotonic": "16968041", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-11-09 12:32:27 UTC", "ConditionTimestampMonotonic": "16968041", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-11-09T12:34:50.062 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2013", "ExecMainStartTimestamp": "Mon 2020-11-09 12:32:34 UTC", "ExecMainStartTimestampMonotonic": "23339171", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-11-09 12:32:27 UTC", "InactiveExitTimestampMonotonic": "16968854", "InvocationID": "8a04d60518cb4cae8c09b945e2e2414d", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-11-09T12:34:50.064 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2013", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-11-09 12:32:34 UTC", "S 2020-11-09T12:34:50.065 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "23339221", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-11-09 12:32:34 UTC", "WatchdogTimestampMonotonic": "23339218", "WatchdogUSec": "0"}} 2020-11-09T12:34:50.097 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:50 +0000 (0:00:00.242) 0:01:29.131 ******* 2020-11-09T12:34:50.124 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:34:50.126 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-09T12:34:50.158 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:50 +0000 (0:00:00.061) 0:01:29.192 ******* 2020-11-09T12:34:50.186 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:34:50.187 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:50.220 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:50 +0000 (0:00:00.062) 0:01:29.254 ******* 2020-11-09T12:34:50.247 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-09T12:34:50.248 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:50.280 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:50 +0000 (0:00:00.060) 0:01:29.314 ******* 2020-11-09T12:34:50.412 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi164.front.sepia.ceph.com] => {"cache_update_time": 1604925189, "cache_updated": false, "changed": false} 2020-11-09T12:34:50.450 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:50 +0000 (0:00:00.169) 0:01:29.484 ******* 2020-11-09T12:34:50.478 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-11-09T12:34:50.480 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:50.519 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:50 +0000 (0:00:00.068) 0:01:29.552 ******* 2020-11-09T12:34:50.640 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-09T12:34:50.642 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 1234, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-11-09T12:34:50.674 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:50 +0000 (0:00:00.154) 0:01:29.707 ******* 2020-11-09T12:34:51.004 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-11-09T12:34:51.006 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1234, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-09T12:34:51.039 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:51 +0000 (0:00:00.365) 0:01:30.073 ******* 2020-11-09T12:34:51.140 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-11-09T12:34:51.142 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"cache_update_time": 1604925193, "cache_updated": false, "changed": false} 2020-11-09T12:34:51.176 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:51 +0000 (0:00:00.137) 0:01:30.210 ******* 2020-11-09T12:34:51.205 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-11-09T12:34:51.207 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:51.240 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:51 +0000 (0:00:00.063) 0:01:30.274 ******* 2020-11-09T12:34:51.376 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "gid": 1234, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-11-09T12:34:51.427 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:51 +0000 (0:00:00.187) 0:01:30.461 ******* 2020-11-09T12:34:51.793 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-11-09T12:34:51.795 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1234, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-09T12:34:51.838 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:51 +0000 (0:00:00.410) 0:01:30.872 ******* 2020-11-09T12:34:52.097 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-11-09T12:34:52.099 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-09T12:34:52.141 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:52 +0000 (0:00:00.303) 0:01:31.175 ******* 2020-11-09T12:34:52.609 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-09T12:34:52.644 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:52 +0000 (0:00:00.503) 0:01:31.678 ******* 2020-11-09T12:34:52.652 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-11-09T12:34:52.653 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false} 2020-11-09T12:34:52.688 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:52 +0000 (0:00:00.043) 0:01:31.722 ******* 2020-11-09T12:34:52.716 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-09T12:34:52.718 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-09T12:34:52.753 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:52 +0000 (0:00:00.064) 0:01:31.787 ******* 2020-11-09T12:34:52.901 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-11-09T12:34:52.903 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1604925175.504098, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1604925175.500096, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 8056, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1604925175.500096, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3017472389", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-11-09T12:34:52.938 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:52 +0000 (0:00:00.185) 0:01:31.972 ******* 2020-11-09T12:34:53.145 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-11-09T12:34:53.146 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false} 2020-11-09T12:34:53.180 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:53 +0000 (0:00:00.242) 0:01:32.214 ******* 2020-11-09T12:34:53.212 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-09T12:34:53.214 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-09T12:34:53.246 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:53 +0000 (0:00:00.066) 0:01:32.280 ******* 2020-11-09T12:34:53.336 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-11-09T12:34:53.337 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-11-09T12:34:53.379 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:53 +0000 (0:00:00.132) 0:01:32.413 ******* 2020-11-09T12:34:53.399 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-11-09T12:34:53.401 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1604925168.7455673, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1604925168.7415655, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 8056, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1604925168.7415655, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3017472389", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-11-09T12:34:53.436 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:53 +0000 (0:00:00.056) 0:01:32.470 ******* 2020-11-09T12:34:53.443 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-09T12:34:53.445 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-09T12:34:53.489 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:53 +0000 (0:00:00.052) 0:01:32.523 ******* 2020-11-09T12:34:53.516 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:53.551 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:53 +0000 (0:00:00.061) 0:01:32.584 ******* 2020-11-09T12:34:53.559 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-11-09T12:34:53.561 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-11-09T12:34:53.595 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:53 +0000 (0:00:00.044) 0:01:32.629 ******* 2020-11-09T12:34:53.623 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-09T12:34:53.626 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-09T12:34:53.663 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:53 +0000 (0:00:00.067) 0:01:32.697 ******* 2020-11-09T12:34:53.680 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-11-09T12:34:53.682 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-09T12:34:53.714 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:53 +0000 (0:00:00.051) 0:01:32.748 ******* 2020-11-09T12:34:53.722 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-11-09T12:34:53.724 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:53.757 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:53 +0000 (0:00:00.042) 0:01:32.791 ******* 2020-11-09T12:34:53.849 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-09T12:34:53.851 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-09T12:34:53.887 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:53 +0000 (0:00:00.129) 0:01:32.921 ******* 2020-11-09T12:34:53.896 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-11-09T12:34:53.897 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-09T12:34:53.933 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:53 +0000 (0:00:00.046) 0:01:32.967 ******* 2020-11-09T12:34:53.941 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-11-09T12:34:53.942 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:53.975 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:53 +0000 (0:00:00.042) 0:01:33.009 ******* 2020-11-09T12:34:54.014 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-11-09T12:34:54.016 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.026 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.028 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.066 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.090) 0:01:33.099 ******* 2020-11-09T12:34:54.073 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-09T12:34:54.075 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-09T12:34:54.109 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.043) 0:01:33.143 ******* 2020-11-09T12:34:54.117 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-11-09T12:34:54.118 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.129 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.131 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.141 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.143 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.179 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.068) 0:01:33.212 ******* 2020-11-09T12:34:54.211 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.033) 0:01:33.245 ******* 2020-11-09T12:34:54.218 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:34:54.219 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.257 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.045) 0:01:33.291 ******* 2020-11-09T12:34:54.264 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-11-09T12:34:54.266 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.266 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.267 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.314 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.055) 0:01:33.346 ******* 2020-11-09T12:34:54.319 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-11-09T12:34:54.321 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.358 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.045) 0:01:33.392 ******* 2020-11-09T12:34:54.365 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-11-09T12:34:54.366 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.368 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.379 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.382 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi190.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.420 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.061) 0:01:33.454 ******* 2020-11-09T12:34:54.427 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-11-09T12:34:54.429 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.465 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.044) 0:01:33.498 ******* 2020-11-09T12:34:54.472 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-09T12:34:54.474 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.510 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.045) 0:01:33.544 ******* 2020-11-09T12:34:54.519 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-11-09T12:34:54.520 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.562 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.052) 0:01:33.596 ******* 2020-11-09T12:34:54.571 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-11-09T12:34:54.573 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.619 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.056) 0:01:33.652 ******* 2020-11-09T12:34:54.626 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-11-09T12:34:54.628 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.670 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.051) 0:01:33.704 ******* 2020-11-09T12:34:54.677 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-11-09T12:34:54.679 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.716 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.045) 0:01:33.749 ******* 2020-11-09T12:34:54.722 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-09T12:34:54.724 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.762 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.046) 0:01:33.796 ******* 2020-11-09T12:34:54.771 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.806 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.044) 0:01:33.840 ******* 2020-11-09T12:34:54.814 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-11-09T12:34:54.816 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.853 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.046) 0:01:33.886 ******* 2020-11-09T12:34:54.860 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.897 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.044) 0:01:33.931 ******* 2020-11-09T12:34:54.906 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.943 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.045) 0:01:33.977 ******* 2020-11-09T12:34:54.951 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-09T12:34:54.952 INFO:teuthology.task.ansible.out:skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:54.993 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:54 +0000 (0:00:00.049) 0:01:34.027 ******* 2020-11-09T12:34:55.180 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:55.371 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:55 +0000 (0:00:00.378) 0:01:34.405 ******* 2020-11-09T12:34:55.378 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-09T12:34:55.379 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011062", "end": "2020-11-09 12:34:54.653338", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-09 12:34:54.642276", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:55.393 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006096", "end": "2020-11-09 12:34:54.950752", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-09 12:34:54.944656", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:55.415 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi190.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-09T12:34:55.453 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:55 +0000 (0:00:00.081) 0:01:34.486 ******* 2020-11-09T12:34:55.524 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-09T12:34:55.527 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005554", "end": "2020-11-09 12:34:55.065407", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-09 12:34:55.059853", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:55.631 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011168", "end": "2020-11-09 12:34:55.532542", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-09 12:34:55.521374", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:55.686 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.034952", "end": "2020-11-09 12:34:55.236564", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-09 12:34:55.201612", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:55.729 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:55 +0000 (0:00:00.275) 0:01:34.762 ******* 2020-11-09T12:34:55.756 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-09T12:34:55.758 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi190.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006039", "end": "2020-11-09 12:34:55.663486", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-09 12:34:55.657447", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:55.791 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:55 +0000 (0:00:00.062) 0:01:34.825 ******* 2020-11-09T12:34:55.881 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005543", "end": "2020-11-09 12:34:55.782548", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-09 12:34:55.777005", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:55.914 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-09T12:34:55.916 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-09T12:34:56.083 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-09T12:34:56.083 INFO:teuthology.task.ansible.out:ok: [smithi190.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.098909", "end": "2020-11-09 12:34:55.992959", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-09 12:34:55.894050", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-09T12:34:56.117 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:56 +0000 (0:00:00.326) 0:01:35.151 ******* 2020-11-09T12:34:56.144 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-09T12:34:56.178 INFO:teuthology.task.ansible.out:Monday 09 November 2020 12:34:56 +0000 (0:00:00.060) 0:01:35.211 ******* 2020-11-09T12:34:56.308 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-09T12:34:56.310 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-09T12:34:56.400 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-11-09T12:34:56.406 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-11-09T12:34:56.409 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-11-09T12:34:56.411 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi164.front.sepia.ceph.com : ok=108 changed=26 unreachable=0 failed=0 skipped=181 rescued=0 ignored=0 smithi190.front.sepia.ceph.com : ok=108 changed=26 unreachable=0 failed=0 skipped=181 rescued=0 ignored=0 Monday 09 November 2020 12:34:56 +0000 (0:00:00.232) 0:01:35.444 ******* =============================================================================== testnode : Ensure packages are not present. ----------------------------- 9.97s testnode : Zap all non-root disks --------------------------------------- 8.61s testnode : Install packages --------------------------------------------- 8.28s testnode : Update apt cache. -------------------------------------------- 2.87s testnode : Install apt keys --------------------------------------------- 2.17s common : Update apt cache ----------------------------------------------- 1.99s Gathering Facts --------------------------------------------------------- 1.81s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.63s testnode : Create logical volume(s) ------------------------------------- 1.48s testnode : Install python-apt ------------------------------------------- 0.90s common : Make sure smartmontools is installed --------------------------- 0.89s Gathering Facts --------------------------------------------------------- 0.85s testnode : Make sure apt dependencies are installed --------------------- 0.74s testnode : Upgrade packages --------------------------------------------- 0.67s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.64s ansible-managed : Add authorized keys for the ansible user. ------------- 0.54s testnode : Install non aarch64 packages --------------------------------- 0.47s ansible-managed : Create the ansible user. ------------------------------ 0.46s testnode : Add ssh pubkeys ------------------- 2020-11-09T12:34:56.413 INFO:teuthology.task.ansible.out:--------------------------- 0.45s testnode : Create volume group(s) --------------------------------------- 0.45s 2020-11-09T12:34:56.738 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi164.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2020-11-09T12:34:56.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-09T12:34:56.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:34:56.821 INFO:teuthology.orchestra.run.smithi164:> true 2020-11-09T12:34:57.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-09T12:34:57.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-09T12:34:57.086 INFO:teuthology.orchestra.run.smithi190:> true 2020-11-09T12:34:57.291 INFO:teuthology.run_tasks:Running task clock... 2020-11-09T12:34:57.303 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-11-09T12:34:57.304 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-09T12:34:57.304 INFO:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-09T12:34:57.373 INFO:teuthology.orchestra.run.smithi164.stdout: 9 Nov 12:34:56 ntpd[6874]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-09T12:34:57.374 INFO:teuthology.orchestra.run.smithi164.stdout: 9 Nov 12:34:56 ntpd[6874]: Command line: ntpd -gq 2020-11-09T12:34:57.374 INFO:teuthology.orchestra.run.smithi164.stdout: 9 Nov 12:34:56 ntpd[6874]: proto: precision = 0.123 usec (-23) 2020-11-09T12:34:57.374 INFO:teuthology.orchestra.run.smithi164.stdout: 9 Nov 12:34:56 ntpd[6874]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-09T12:34:57.374 INFO:teuthology.orchestra.run.smithi164.stdout: 9 Nov 12:34:56 ntpd[6874]: restrict ::: KOD does nothing without LIMITED. 2020-11-09T12:34:57.375 INFO:teuthology.orchestra.run.smithi164.stdout: 9 Nov 12:34:56 ntpd[6874]: Listen and drop on 0 v6wildcard [::]:123 2020-11-09T12:34:57.375 INFO:teuthology.orchestra.run.smithi164.stdout: 9 Nov 12:34:56 ntpd[6874]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-09T12:34:57.375 INFO:teuthology.orchestra.run.smithi164.stdout: 9 Nov 12:34:56 ntpd[6874]: Listen normally on 2 lo 127.0.0.1:123 2020-11-09T12:34:57.375 INFO:teuthology.orchestra.run.smithi164.stdout: 9 Nov 12:34:56 ntpd[6874]: Listen normally on 3 enp3s0f1 172.21.15.164:123 2020-11-09T12:34:57.375 INFO:teuthology.orchestra.run.smithi164.stdout: 9 Nov 12:34:56 ntpd[6874]: Listen normally on 4 lo [::1]:123 2020-11-09T12:34:57.375 INFO:teuthology.orchestra.run.smithi164.stdout: 9 Nov 12:34:56 ntpd[6874]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:73f3%5]:123 2020-11-09T12:34:57.375 INFO:teuthology.orchestra.run.smithi164.stdout: 9 Nov 12:34:56 ntpd[6874]: Listening on routing socket on fd #22 for interface updates 2020-11-09T12:34:57.376 INFO:teuthology.orchestra.run.smithi164.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-09T12:34:57.376 INFO:teuthology.orchestra.run.smithi164.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-09T12:34:58.370 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:57 ntpd[6874]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:34:58.371 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:57 ntpd[6874]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:34:58.371 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:57 ntpd[6874]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:34:58.371 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:57 ntpd[6874]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:34:58.371 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:57 ntpd[6874]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:34:58.371 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:57 ntpd[6874]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:34:58.371 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:57 ntpd[6874]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:34:58.371 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:57 ntpd[6874]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:00.369 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:59 ntpd[6874]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:00.369 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:59 ntpd[6874]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:00.369 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:59 ntpd[6874]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:00.370 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:59 ntpd[6874]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:00.370 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:59 ntpd[6874]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:00.370 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:59 ntpd[6874]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:00.370 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:59 ntpd[6874]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:00.370 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:59 ntpd[6874]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:00.370 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:59 ntpd[6874]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:00.371 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:59 ntpd[6874]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:00.371 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:59 ntpd[6874]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:00.371 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:34:59 ntpd[6874]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:02.369 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:02.369 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:02.369 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:02.369 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:02.369 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:02.370 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:02.370 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:02.370 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:02.370 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:02.370 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:02.370 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:02.370 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:02.371 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:02.371 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:02.371 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:02.371 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:01 ntpd[6874]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:04.367 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:03 ntpd[6874]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:04.368 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:03 ntpd[6874]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:04.368 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:03 ntpd[6874]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:04.368 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:03 ntpd[6874]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:04.368 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:03 ntpd[6874]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:04.369 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:03 ntpd[6874]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:04.369 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:03 ntpd[6874]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:04.369 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:03 ntpd[6874]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:04.369 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:04 ntpd[6874]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-09T12:35:04.369 INFO:teuthology.orchestra.run.smithi164.stderr: 9 Nov 12:35:04 ntpd[6874]: can't open /var/log/ntpstats/loopstats.20201109: Permission denied 2020-11-09T12:35:04.369 INFO:teuthology.orchestra.run.smithi164.stdout: 9 Nov 12:35:04 ntpd[6874]: ntpd: time set +0.415308 s 2020-11-09T12:35:04.370 INFO:teuthology.orchestra.run.smithi164.stdout:ntpd: time set +0.415308s 2020-11-09T12:35:04.421 INFO:teuthology.orchestra.run.smithi164.stdout: remote refid st t when poll reach delay offset jitter 2020-11-09T12:35:04.421 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================== 2020-11-09T12:35:04.422 INFO:teuthology.orchestra.run.smithi164.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T12:35:04.422 INFO:teuthology.orchestra.run.smithi164.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T12:35:04.422 INFO:teuthology.orchestra.run.smithi164.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T12:35:04.422 INFO:teuthology.orchestra.run.smithi164.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T12:35:04.423 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-09T12:35:04.424 INFO:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-09T12:35:04.474 INFO:teuthology.orchestra.run.smithi190.stdout: 9 Nov 12:35:04 ntpd[6942]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-09T12:35:04.475 INFO:teuthology.orchestra.run.smithi190.stdout: 9 Nov 12:35:04 ntpd[6942]: Command line: ntpd -gq 2020-11-09T12:35:04.476 INFO:teuthology.orchestra.run.smithi190.stdout: 9 Nov 12:35:04 ntpd[6942]: proto: precision = 0.098 usec (-23) 2020-11-09T12:35:04.476 INFO:teuthology.orchestra.run.smithi190.stdout: 9 Nov 12:35:04 ntpd[6942]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-09T12:35:04.476 INFO:teuthology.orchestra.run.smithi190.stdout: 9 Nov 12:35:04 ntpd[6942]: restrict ::: KOD does nothing without LIMITED. 2020-11-09T12:35:04.476 INFO:teuthology.orchestra.run.smithi190.stdout: 9 Nov 12:35:04 ntpd[6942]: Listen and drop on 0 v6wildcard [::]:123 2020-11-09T12:35:04.476 INFO:teuthology.orchestra.run.smithi190.stdout: 9 Nov 12:35:04 ntpd[6942]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-09T12:35:04.477 INFO:teuthology.orchestra.run.smithi190.stdout: 9 Nov 12:35:04 ntpd[6942]: Listen normally on 2 lo 127.0.0.1:123 2020-11-09T12:35:04.477 INFO:teuthology.orchestra.run.smithi190.stdout: 9 Nov 12:35:04 ntpd[6942]: Listen normally on 3 enp3s0f1 172.21.15.190:123 2020-11-09T12:35:04.477 INFO:teuthology.orchestra.run.smithi190.stdout: 9 Nov 12:35:04 ntpd[6942]: Listen normally on 4 lo [::1]:123 2020-11-09T12:35:04.477 INFO:teuthology.orchestra.run.smithi190.stdout: 9 Nov 12:35:04 ntpd[6942]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7763%5]:123 2020-11-09T12:35:04.477 INFO:teuthology.orchestra.run.smithi190.stdout: 9 Nov 12:35:04 ntpd[6942]: Listening on routing socket on fd #22 for interface updates 2020-11-09T12:35:04.477 INFO:teuthology.orchestra.run.smithi190.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-09T12:35:04.477 INFO:teuthology.orchestra.run.smithi190.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-09T12:35:05.476 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:05 ntpd[6942]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:05.476 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:05 ntpd[6942]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:05.476 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:05 ntpd[6942]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:05.477 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:05 ntpd[6942]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:05.477 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:05 ntpd[6942]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:05.477 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:05 ntpd[6942]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:05.478 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:05 ntpd[6942]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:05.478 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:05 ntpd[6942]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:05.478 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:05 ntpd[6942]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:05.479 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:05 ntpd[6942]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:05.479 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:05 ntpd[6942]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:05.479 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:05 ntpd[6942]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:07.474 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:07 ntpd[6942]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:07.474 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:07 ntpd[6942]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:07.474 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:07 ntpd[6942]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:07.475 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:07 ntpd[6942]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:07.475 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:07 ntpd[6942]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:07.475 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:07 ntpd[6942]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:07.475 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:07 ntpd[6942]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:07.475 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:07 ntpd[6942]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:07.475 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:07 ntpd[6942]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:07.475 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:07 ntpd[6942]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:07.476 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:07 ntpd[6942]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:07.476 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:07 ntpd[6942]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:09.473 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:09.473 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:09.474 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:09.474 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:09.474 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:09.474 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:09.474 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:09.474 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:09.474 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:09.475 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:09.475 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:09.475 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:09.475 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:09.475 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:09.475 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:09.475 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:09 ntpd[6942]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:11.472 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:11 ntpd[6942]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-09T12:35:11.472 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:11 ntpd[6942]: can't open /var/log/ntpstats/rawstats.20201109: Permission denied 2020-11-09T12:35:11.472 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:11 ntpd[6942]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-09T12:35:11.472 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:11 ntpd[6942]: can't open /var/log/ntpstats/peerstats.20201109: Permission denied 2020-11-09T12:35:11.472 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:11 ntpd[6942]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-09T12:35:11.472 INFO:teuthology.orchestra.run.smithi190.stderr: 9 Nov 12:35:11 ntpd[6942]: can't open /var/log/ntpstats/loopstats.20201109: Permission denied 2020-11-09T12:35:11.473 INFO:teuthology.orchestra.run.smithi190.stdout: 9 Nov 12:35:11 ntpd[6942]: ntpd: time set +0.057615 s 2020-11-09T12:35:11.473 INFO:teuthology.orchestra.run.smithi190.stdout:ntpd: time set +0.057615s 2020-11-09T12:35:11.534 INFO:teuthology.orchestra.run.smithi190.stdout: remote refid st t when poll reach delay offset jitter 2020-11-09T12:35:11.534 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================== 2020-11-09T12:35:11.534 INFO:teuthology.orchestra.run.smithi190.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T12:35:11.534 INFO:teuthology.orchestra.run.smithi190.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T12:35:11.534 INFO:teuthology.orchestra.run.smithi190.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T12:35:11.534 INFO:teuthology.orchestra.run.smithi190.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-09T12:35:11.535 INFO:teuthology.run_tasks:Running task install... 2020-11-09T12:35:11.546 DEBUG:teuthology.task.install:project ceph 2020-11-09T12:35:11.546 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d'}} 2020-11-09T12:35:11.547 DEBUG:teuthology.task.install:config {'sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d'} 2020-11-09T12:35:11.547 INFO:teuthology.task.install:Using flavor: basic 2020-11-09T12:35:11.553 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-11-09T12:35:11.554 INFO:teuthology.task.install:extra packages: [] 2020-11-09T12:35:11.554 INFO:teuthology.orchestra.run.smithi164:> sudo lsb_release -is 2020-11-09T12:35:11.606 INFO:teuthology.orchestra.run.smithi164.stdout:Ubuntu 2020-11-09T12:35:11.611 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-09T12:35:11.612 INFO:teuthology.orchestra.run.smithi190:> sudo lsb_release -is 2020-11-09T12:35:11.613 INFO:teuthology.orchestra.run.smithi164:> sudo apt-key list | grep Ceph 2020-11-09T12:35:11.664 INFO:teuthology.orchestra.run.smithi190.stdout:Ubuntu 2020-11-09T12:35:11.670 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-09T12:35:11.670 INFO:teuthology.orchestra.run.smithi190:> sudo apt-key list | grep Ceph 2020-11-09T12:35:11.705 INFO:teuthology.orchestra.run.smithi164.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-09T12:35:11.759 INFO:teuthology.orchestra.run.smithi190.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-09T12:35:11.761 INFO:teuthology.orchestra.run.smithi164.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-09T12:35:11.761 INFO:teuthology.orchestra.run.smithi164.stdout:uid [ unknown] Ceph.com (release key) 2020-11-09T12:35:11.762 INFO:teuthology.orchestra.run.smithi164:> sudo lsb_release -is 2020-11-09T12:35:11.806 INFO:teuthology.orchestra.run.smithi164.stdout:Ubuntu 2020-11-09T12:35:11.810 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-09T12:35:11.811 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-11-09T12:35:11.811 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=afece648f8ccac358b59a7b305a02d8c55dc908d 2020-11-09T12:35:11.816 INFO:teuthology.orchestra.run.smithi190.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-09T12:35:11.816 INFO:teuthology.orchestra.run.smithi190.stdout:uid [ unknown] Ceph.com (release key) 2020-11-09T12:35:11.817 INFO:teuthology.orchestra.run.smithi190:> sudo lsb_release -is 2020-11-09T12:35:11.868 INFO:teuthology.orchestra.run.smithi190.stdout:Ubuntu 2020-11-09T12:35:11.874 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-09T12:35:11.875 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-11-09T12:35:11.875 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=afece648f8ccac358b59a7b305a02d8c55dc908d 2020-11-09T12:35:11.984 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default/ 2020-11-09T12:35:11.985 INFO:teuthology.task.install.deb:Package version is 16.0.0-7008-gafece648-1bionic 2020-11-09T12:35:12.034 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default/ 2020-11-09T12:35:12.035 INFO:teuthology.task.install.deb:Package version is 16.0.0-7008-gafece648-1bionic 2020-11-09T12:35:12.135 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:35:12.136 INFO:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-09T12:35:12.142 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-09T12:35:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:35:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:35:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:170 bytes copied, 7.196e-05 s, 2.4 MB/s 2020-11-09T12:35:12.153 INFO:teuthology.orchestra.run.smithi164:> sudo apt-get update 2020-11-09T12:35:12.166 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:35:12.167 INFO:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-09T12:35:12.172 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-09T12:35:12.181 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:35:12.182 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:35:12.182 INFO:teuthology.orchestra.run.smithi190.stderr:170 bytes copied, 7.5627e-05 s, 2.2 MB/s 2020-11-09T12:35:12.183 INFO:teuthology.orchestra.run.smithi190:> sudo apt-get update 2020-11-09T12:35:12.335 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-09T12:35:12.369 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-09T12:35:12.440 INFO:teuthology.orchestra.run.smithi164.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic InRelease 2020-11-09T12:35:12.447 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-09T12:35:12.464 INFO:teuthology.orchestra.run.smithi164.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-09T12:35:12.471 INFO:teuthology.orchestra.run.smithi190.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic InRelease 2020-11-09T12:35:12.485 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-09T12:35:12.489 INFO:teuthology.orchestra.run.smithi164.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-09T12:35:12.494 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-09T12:35:12.512 INFO:teuthology.orchestra.run.smithi164.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,953 B] 2020-11-09T12:35:12.519 INFO:teuthology.orchestra.run.smithi190.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-09T12:35:12.525 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-09T12:35:12.543 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-09T12:35:12.549 INFO:teuthology.orchestra.run.smithi164.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-09T12:35:12.565 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-09T12:35:12.582 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,953 B] 2020-11-09T12:35:12.605 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-09T12:35:12.645 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-09T12:35:12.941 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 26.7 kB in 1s (45.6 kB/s) 2020-11-09T12:35:12.983 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 26.7 kB in 1s (44.9 kB/s) 2020-11-09T12:35:14.156 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2020-11-09T12:35:14.203 INFO:teuthology.orchestra.run.smithi164:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-7008-gafece648-1bionic cephadm=16.0.0-7008-gafece648-1bionic ceph-mds=16.0.0-7008-gafece648-1bionic ceph-mgr=16.0.0-7008-gafece648-1bionic ceph-common=16.0.0-7008-gafece648-1bionic ceph-fuse=16.0.0-7008-gafece648-1bionic ceph-test=16.0.0-7008-gafece648-1bionic radosgw=16.0.0-7008-gafece648-1bionic python3-rados=16.0.0-7008-gafece648-1bionic python3-rgw=16.0.0-7008-gafece648-1bionic python3-cephfs=16.0.0-7008-gafece648-1bionic python3-rbd=16.0.0-7008-gafece648-1bionic libcephfs2=16.0.0-7008-gafece648-1bionic libcephfs-dev=16.0.0-7008-gafece648-1bionic librados2=16.0.0-7008-gafece648-1bionic librbd1=16.0.0-7008-gafece648-1bionic rbd-fuse=16.0.0-7008-gafece648-1bionic 2020-11-09T12:35:14.204 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2020-11-09T12:35:14.252 INFO:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-7008-gafece648-1bionic cephadm=16.0.0-7008-gafece648-1bionic ceph-mds=16.0.0-7008-gafece648-1bionic ceph-mgr=16.0.0-7008-gafece648-1bionic ceph-common=16.0.0-7008-gafece648-1bionic ceph-fuse=16.0.0-7008-gafece648-1bionic ceph-test=16.0.0-7008-gafece648-1bionic radosgw=16.0.0-7008-gafece648-1bionic python3-rados=16.0.0-7008-gafece648-1bionic python3-rgw=16.0.0-7008-gafece648-1bionic python3-cephfs=16.0.0-7008-gafece648-1bionic python3-rbd=16.0.0-7008-gafece648-1bionic libcephfs2=16.0.0-7008-gafece648-1bionic libcephfs-dev=16.0.0-7008-gafece648-1bionic librados2=16.0.0-7008-gafece648-1bionic librbd1=16.0.0-7008-gafece648-1bionic rbd-fuse=16.0.0-7008-gafece648-1bionic 2020-11-09T12:35:14.287 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2020-11-09T12:35:14.333 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2020-11-09T12:35:14.543 INFO:teuthology.orchestra.run.smithi164.stdout:Building dependency tree... 2020-11-09T12:35:14.544 INFO:teuthology.orchestra.run.smithi164.stdout:Reading state information... 2020-11-09T12:35:14.589 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2020-11-09T12:35:14.589 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2020-11-09T12:35:14.710 INFO:teuthology.orchestra.run.smithi164.stdout:The following additional packages will be installed: 2020-11-09T12:35:14.710 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-09T12:35:14.711 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-09T12:35:14.712 INFO:teuthology.orchestra.run.smithi164.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-09T12:35:14.712 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-09T12:35:14.712 INFO:teuthology.orchestra.run.smithi164.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-09T12:35:14.712 INFO:teuthology.orchestra.run.smithi164.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-09T12:35:14.712 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-09T12:35:14.712 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-11-09T12:35:14.713 INFO:teuthology.orchestra.run.smithi164.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-11-09T12:35:14.713 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-11-09T12:35:14.713 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-09T12:35:14.713 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-09T12:35:14.713 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-09T12:35:14.713 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-09T12:35:14.713 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-09T12:35:14.713 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-11-09T12:35:14.714 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-09T12:35:14.714 INFO:teuthology.orchestra.run.smithi164.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-09T12:35:14.714 INFO:teuthology.orchestra.run.smithi164.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-09T12:35:14.714 INFO:teuthology.orchestra.run.smithi164.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-09T12:35:14.714 INFO:teuthology.orchestra.run.smithi164.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-09T12:35:14.714 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-09T12:35:14.715 INFO:teuthology.orchestra.run.smithi164.stdout:Suggested packages: 2020-11-09T12:35:14.715 INFO:teuthology.orchestra.run.smithi164.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-09T12:35:14.715 INFO:teuthology.orchestra.run.smithi164.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-09T12:35:14.715 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-09T12:35:14.715 INFO:teuthology.orchestra.run.smithi164.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-09T12:35:14.716 INFO:teuthology.orchestra.run.smithi164.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-09T12:35:14.716 INFO:teuthology.orchestra.run.smithi164.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-09T12:35:14.716 INFO:teuthology.orchestra.run.smithi164.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-09T12:35:14.716 INFO:teuthology.orchestra.run.smithi164.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-09T12:35:14.716 INFO:teuthology.orchestra.run.smithi164.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-09T12:35:14.716 INFO:teuthology.orchestra.run.smithi164.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-09T12:35:14.716 INFO:teuthology.orchestra.run.smithi164.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-09T12:35:14.717 INFO:teuthology.orchestra.run.smithi164.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-09T12:35:14.717 INFO:teuthology.orchestra.run.smithi164.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-09T12:35:14.717 INFO:teuthology.orchestra.run.smithi164.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-09T12:35:14.717 INFO:teuthology.orchestra.run.smithi164.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-09T12:35:14.751 INFO:teuthology.orchestra.run.smithi190.stdout:The following additional packages will be installed: 2020-11-09T12:35:14.751 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-09T12:35:14.752 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-09T12:35:14.752 INFO:teuthology.orchestra.run.smithi190.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-09T12:35:14.752 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-09T12:35:14.752 INFO:teuthology.orchestra.run.smithi190.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-09T12:35:14.753 INFO:teuthology.orchestra.run.smithi190.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-09T12:35:14.753 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-09T12:35:14.753 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-11-09T12:35:14.753 INFO:teuthology.orchestra.run.smithi190.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-11-09T12:35:14.753 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-11-09T12:35:14.753 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-09T12:35:14.753 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-09T12:35:14.753 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-09T12:35:14.754 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-09T12:35:14.754 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-09T12:35:14.754 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-11-09T12:35:14.754 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-09T12:35:14.754 INFO:teuthology.orchestra.run.smithi190.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-09T12:35:14.754 INFO:teuthology.orchestra.run.smithi190.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-09T12:35:14.754 INFO:teuthology.orchestra.run.smithi190.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-09T12:35:14.755 INFO:teuthology.orchestra.run.smithi190.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-09T12:35:14.755 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-09T12:35:14.755 INFO:teuthology.orchestra.run.smithi190.stdout:Suggested packages: 2020-11-09T12:35:14.755 INFO:teuthology.orchestra.run.smithi190.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-09T12:35:14.755 INFO:teuthology.orchestra.run.smithi190.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-09T12:35:14.756 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-09T12:35:14.756 INFO:teuthology.orchestra.run.smithi190.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-09T12:35:14.756 INFO:teuthology.orchestra.run.smithi190.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-09T12:35:14.756 INFO:teuthology.orchestra.run.smithi190.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-09T12:35:14.756 INFO:teuthology.orchestra.run.smithi190.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-09T12:35:14.756 INFO:teuthology.orchestra.run.smithi190.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-09T12:35:14.756 INFO:teuthology.orchestra.run.smithi190.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-09T12:35:14.757 INFO:teuthology.orchestra.run.smithi190.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-09T12:35:14.757 INFO:teuthology.orchestra.run.smithi190.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-09T12:35:14.757 INFO:teuthology.orchestra.run.smithi190.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-09T12:35:14.757 INFO:teuthology.orchestra.run.smithi190.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-09T12:35:14.757 INFO:teuthology.orchestra.run.smithi190.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-09T12:35:14.757 INFO:teuthology.orchestra.run.smithi190.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-09T12:35:14.809 INFO:teuthology.orchestra.run.smithi164.stdout:The following NEW packages will be installed: 2020-11-09T12:35:14.810 INFO:teuthology.orchestra.run.smithi164.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-09T12:35:14.810 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-09T12:35:14.811 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-11-09T12:35:14.811 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-11-09T12:35:14.811 INFO:teuthology.orchestra.run.smithi164.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-11-09T12:35:14.811 INFO:teuthology.orchestra.run.smithi164.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-11-09T12:35:14.811 INFO:teuthology.orchestra.run.smithi164.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-11-09T12:35:14.812 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-11-09T12:35:14.812 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-11-09T12:35:14.812 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-09T12:35:14.812 INFO:teuthology.orchestra.run.smithi164.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-09T12:35:14.812 INFO:teuthology.orchestra.run.smithi164.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-09T12:35:14.812 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-09T12:35:14.812 INFO:teuthology.orchestra.run.smithi164.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-09T12:35:14.812 INFO:teuthology.orchestra.run.smithi164.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-09T12:35:14.813 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-09T12:35:14.813 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-09T12:35:14.813 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-11-09T12:35:14.813 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-11-09T12:35:14.813 INFO:teuthology.orchestra.run.smithi164.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-09T12:35:14.813 INFO:teuthology.orchestra.run.smithi164.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-09T12:35:14.813 INFO:teuthology.orchestra.run.smithi164.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-09T12:35:14.814 INFO:teuthology.orchestra.run.smithi164.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-09T12:35:14.814 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-09T12:35:14.814 INFO:teuthology.orchestra.run.smithi164.stdout:The following packages will be upgraded: 2020-11-09T12:35:14.814 INFO:teuthology.orchestra.run.smithi164.stdout: librados2 librbd1 2020-11-09T12:35:14.851 INFO:teuthology.orchestra.run.smithi190.stdout:The following NEW packages will be installed: 2020-11-09T12:35:14.852 INFO:teuthology.orchestra.run.smithi190.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-09T12:35:14.852 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-09T12:35:14.852 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-11-09T12:35:14.853 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-11-09T12:35:14.853 INFO:teuthology.orchestra.run.smithi190.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-11-09T12:35:14.853 INFO:teuthology.orchestra.run.smithi190.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-11-09T12:35:14.853 INFO:teuthology.orchestra.run.smithi190.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-11-09T12:35:14.854 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-11-09T12:35:14.854 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-11-09T12:35:14.854 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-09T12:35:14.854 INFO:teuthology.orchestra.run.smithi190.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-09T12:35:14.854 INFO:teuthology.orchestra.run.smithi190.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-09T12:35:14.854 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-09T12:35:14.854 INFO:teuthology.orchestra.run.smithi190.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-09T12:35:14.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-09T12:35:14.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-09T12:35:14.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-09T12:35:14.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-11-09T12:35:14.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-11-09T12:35:14.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-09T12:35:14.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-09T12:35:14.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-09T12:35:14.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-09T12:35:14.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-09T12:35:14.856 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages will be upgraded: 2020-11-09T12:35:14.857 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 librbd1 2020-11-09T12:35:14.977 INFO:teuthology.orchestra.run.smithi164.stdout:2 upgraded, 110 newly installed, 0 to remove and 0 not upgraded. 2020-11-09T12:35:14.977 INFO:teuthology.orchestra.run.smithi164.stdout:Need to get 168 MB of archives. 2020-11-09T12:35:14.977 INFO:teuthology.orchestra.run.smithi164.stdout:After this operation, 782 MB of additional disk space will be used. 2020-11-09T12:35:14.977 INFO:teuthology.orchestra.run.smithi164.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-7008-gafece648-1bionic [1,921 kB] 2020-11-09T12:35:14.998 INFO:teuthology.orchestra.run.smithi164.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-09T12:35:15.013 INFO:teuthology.orchestra.run.smithi190.stdout:2 upgraded, 110 newly installed, 0 to remove and 0 not upgraded. 2020-11-09T12:35:15.013 INFO:teuthology.orchestra.run.smithi190.stdout:Need to get 168 MB of archives. 2020-11-09T12:35:15.013 INFO:teuthology.orchestra.run.smithi190.stdout:After this operation, 782 MB of additional disk space will be used. 2020-11-09T12:35:15.013 INFO:teuthology.orchestra.run.smithi190.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-7008-gafece648-1bionic [1,921 kB] 2020-11-09T12:35:15.037 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-09T12:35:15.077 INFO:teuthology.orchestra.run.smithi164.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-09T12:35:15.118 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-09T12:35:15.212 INFO:teuthology.orchestra.run.smithi164.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-7008-gafece648-1bionic [3,157 kB] 2020-11-09T12:35:15.266 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-7008-gafece648-1bionic [3,157 kB] 2020-11-09T12:35:15.335 INFO:teuthology.orchestra.run.smithi164.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-09T12:35:15.336 INFO:teuthology.orchestra.run.smithi164.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-09T12:35:15.338 INFO:teuthology.orchestra.run.smithi164.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-09T12:35:15.355 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-09T12:35:15.355 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-09T12:35:15.359 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-09T12:35:15.403 INFO:teuthology.orchestra.run.smithi164.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-7008-gafece648-1bionic [477 kB] 2020-11-09T12:35:15.431 INFO:teuthology.orchestra.run.smithi164.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-7008-gafece648-1bionic [331 kB] 2020-11-09T12:35:15.446 INFO:teuthology.orchestra.run.smithi164.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-09T12:35:15.449 INFO:teuthology.orchestra.run.smithi164.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-09T12:35:15.455 INFO:teuthology.orchestra.run.smithi164.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-7008-gafece648-1bionic [20.6 kB] 2020-11-09T12:35:15.456 INFO:teuthology.orchestra.run.smithi164.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-7008-gafece648-1bionic [168 kB] 2020-11-09T12:35:15.461 INFO:teuthology.orchestra.run.smithi164.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-7008-gafece648-1bionic [27.7 kB] 2020-11-09T12:35:15.463 INFO:teuthology.orchestra.run.smithi164.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-7008-gafece648-1bionic [330 kB] 2020-11-09T12:35:15.475 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-09T12:35:15.476 INFO:teuthology.orchestra.run.smithi190.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-7008-gafece648-1bionic [477 kB] 2020-11-09T12:35:15.477 INFO:teuthology.orchestra.run.smithi190.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-09T12:35:15.488 INFO:teuthology.orchestra.run.smithi164.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-09T12:35:15.488 INFO:teuthology.orchestra.run.smithi164.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-7008-gafece648-1bionic [3,169 kB] 2020-11-09T12:35:15.495 INFO:teuthology.orchestra.run.smithi164.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-09T12:35:15.503 INFO:teuthology.orchestra.run.smithi164.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-09T12:35:15.508 INFO:teuthology.orchestra.run.smithi190.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-7008-gafece648-1bionic [331 kB] 2020-11-09T12:35:15.515 INFO:teuthology.orchestra.run.smithi164.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-09T12:35:15.520 INFO:teuthology.orchestra.run.smithi190.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-09T12:35:15.524 INFO:teuthology.orchestra.run.smithi164.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-09T12:35:15.527 INFO:teuthology.orchestra.run.smithi190.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-09T12:35:15.537 INFO:teuthology.orchestra.run.smithi190.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-09T12:35:15.537 INFO:teuthology.orchestra.run.smithi190.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-7008-gafece648-1bionic [20.6 kB] 2020-11-09T12:35:15.538 INFO:teuthology.orchestra.run.smithi190.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-7008-gafece648-1bionic [168 kB] 2020-11-09T12:35:15.546 INFO:teuthology.orchestra.run.smithi190.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-7008-gafece648-1bionic [27.7 kB] 2020-11-09T12:35:15.546 INFO:teuthology.orchestra.run.smithi190.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-7008-gafece648-1bionic [330 kB] 2020-11-09T12:35:15.547 INFO:teuthology.orchestra.run.smithi190.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-09T12:35:15.548 INFO:teuthology.orchestra.run.smithi164.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-09T12:35:15.553 INFO:teuthology.orchestra.run.smithi164.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-09T12:35:15.555 INFO:teuthology.orchestra.run.smithi190.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-09T12:35:15.574 INFO:teuthology.orchestra.run.smithi164.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-09T12:35:15.575 INFO:teuthology.orchestra.run.smithi190.stdout:Get:21 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-7008-gafece648-1bionic [3,169 kB] 2020-11-09T12:35:15.576 INFO:teuthology.orchestra.run.smithi190.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-09T12:35:15.586 INFO:teuthology.orchestra.run.smithi164.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-09T12:35:15.586 INFO:teuthology.orchestra.run.smithi164.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-09T12:35:15.588 INFO:teuthology.orchestra.run.smithi164.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-09T12:35:15.590 INFO:teuthology.orchestra.run.smithi190.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-09T12:35:15.595 INFO:teuthology.orchestra.run.smithi190.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-09T12:35:15.614 INFO:teuthology.orchestra.run.smithi190.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-09T12:35:15.652 INFO:teuthology.orchestra.run.smithi164.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-09T12:35:15.655 INFO:teuthology.orchestra.run.smithi164.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-09T12:35:15.655 INFO:teuthology.orchestra.run.smithi164.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-09T12:35:15.656 INFO:teuthology.orchestra.run.smithi164.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-09T12:35:15.659 INFO:teuthology.orchestra.run.smithi164.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-09T12:35:15.661 INFO:teuthology.orchestra.run.smithi164.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-09T12:35:15.669 INFO:teuthology.orchestra.run.smithi190.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-09T12:35:15.670 INFO:teuthology.orchestra.run.smithi190.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-09T12:35:15.670 INFO:teuthology.orchestra.run.smithi190.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-09T12:35:15.673 INFO:teuthology.orchestra.run.smithi190.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-09T12:35:15.674 INFO:teuthology.orchestra.run.smithi190.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-09T12:35:15.675 INFO:teuthology.orchestra.run.smithi190.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-09T12:35:15.677 INFO:teuthology.orchestra.run.smithi190.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-09T12:35:15.678 INFO:teuthology.orchestra.run.smithi164.stdout:Get:34 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-7008-gafece648-1bionic [99.1 kB] 2020-11-09T12:35:15.682 INFO:teuthology.orchestra.run.smithi164.stdout:Get:35 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-7008-gafece648-1bionic [414 kB] 2020-11-09T12:35:15.714 INFO:teuthology.orchestra.run.smithi164.stdout:Get:36 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-7008-gafece648-1bionic [20.5 MB] 2020-11-09T12:35:15.730 INFO:teuthology.orchestra.run.smithi164.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-09T12:35:15.731 INFO:teuthology.orchestra.run.smithi164.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-09T12:35:15.747 INFO:teuthology.orchestra.run.smithi190.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-09T12:35:15.749 INFO:teuthology.orchestra.run.smithi190.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-09T12:35:15.749 INFO:teuthology.orchestra.run.smithi190.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-09T12:35:15.752 INFO:teuthology.orchestra.run.smithi164.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-09T12:35:15.752 INFO:teuthology.orchestra.run.smithi164.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-09T12:35:15.787 INFO:teuthology.orchestra.run.smithi190.stdout:Get:36 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-7008-gafece648-1bionic [99.1 kB] 2020-11-09T12:35:15.791 INFO:teuthology.orchestra.run.smithi190.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-7008-gafece648-1bionic [414 kB] 2020-11-09T12:35:15.808 INFO:teuthology.orchestra.run.smithi164.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-09T12:35:15.810 INFO:teuthology.orchestra.run.smithi164.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-09T12:35:15.812 INFO:teuthology.orchestra.run.smithi164.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-09T12:35:15.821 INFO:teuthology.orchestra.run.smithi164.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-09T12:35:15.825 INFO:teuthology.orchestra.run.smithi190.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-09T12:35:15.825 INFO:teuthology.orchestra.run.smithi190.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-09T12:35:15.826 INFO:teuthology.orchestra.run.smithi164.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-09T12:35:15.826 INFO:teuthology.orchestra.run.smithi164.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-09T12:35:15.826 INFO:teuthology.orchestra.run.smithi190.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-09T12:35:15.827 INFO:teuthology.orchestra.run.smithi190.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-09T12:35:15.830 INFO:teuthology.orchestra.run.smithi190.stdout:Get:42 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-7008-gafece648-1bionic [20.5 MB] 2020-11-09T12:35:15.831 INFO:teuthology.orchestra.run.smithi190.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-09T12:35:15.838 INFO:teuthology.orchestra.run.smithi190.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-09T12:35:15.842 INFO:teuthology.orchestra.run.smithi190.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-09T12:35:15.886 INFO:teuthology.orchestra.run.smithi164.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-09T12:35:15.903 INFO:teuthology.orchestra.run.smithi190.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-09T12:35:15.904 INFO:teuthology.orchestra.run.smithi190.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-09T12:35:15.980 INFO:teuthology.orchestra.run.smithi164.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-09T12:35:15.980 INFO:teuthology.orchestra.run.smithi164.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-09T12:35:15.992 INFO:teuthology.orchestra.run.smithi190.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-09T12:35:15.992 INFO:teuthology.orchestra.run.smithi190.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-09T12:35:16.249 INFO:teuthology.orchestra.run.smithi164.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-09T12:35:16.270 INFO:teuthology.orchestra.run.smithi190.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-09T12:35:16.308 INFO:teuthology.orchestra.run.smithi190.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-09T12:35:16.309 INFO:teuthology.orchestra.run.smithi164.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-09T12:35:16.309 INFO:teuthology.orchestra.run.smithi190.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-09T12:35:16.311 INFO:teuthology.orchestra.run.smithi164.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-09T12:35:16.330 INFO:teuthology.orchestra.run.smithi164.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-09T12:35:16.331 INFO:teuthology.orchestra.run.smithi164.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-09T12:35:16.331 INFO:teuthology.orchestra.run.smithi164.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-09T12:35:16.331 INFO:teuthology.orchestra.run.smithi164.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-09T12:35:16.332 INFO:teuthology.orchestra.run.smithi164.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-09T12:35:16.332 INFO:teuthology.orchestra.run.smithi164.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-09T12:35:16.333 INFO:teuthology.orchestra.run.smithi164.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-09T12:35:16.352 INFO:teuthology.orchestra.run.smithi190.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-09T12:35:16.352 INFO:teuthology.orchestra.run.smithi190.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-09T12:35:16.352 INFO:teuthology.orchestra.run.smithi190.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-09T12:35:16.353 INFO:teuthology.orchestra.run.smithi190.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-09T12:35:16.353 INFO:teuthology.orchestra.run.smithi190.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-09T12:35:16.354 INFO:teuthology.orchestra.run.smithi190.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-09T12:35:16.354 INFO:teuthology.orchestra.run.smithi190.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-09T12:35:16.407 INFO:teuthology.orchestra.run.smithi164.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-09T12:35:16.409 INFO:teuthology.orchestra.run.smithi164.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-09T12:35:16.428 INFO:teuthology.orchestra.run.smithi190.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-09T12:35:16.429 INFO:teuthology.orchestra.run.smithi190.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-09T12:35:16.485 INFO:teuthology.orchestra.run.smithi164.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-09T12:35:16.486 INFO:teuthology.orchestra.run.smithi164.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-09T12:35:16.491 INFO:teuthology.orchestra.run.smithi164.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-09T12:35:16.491 INFO:teuthology.orchestra.run.smithi164.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-09T12:35:16.492 INFO:teuthology.orchestra.run.smithi164.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-09T12:35:16.497 INFO:teuthology.orchestra.run.smithi164.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-09T12:35:16.498 INFO:teuthology.orchestra.run.smithi164.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-09T12:35:16.498 INFO:teuthology.orchestra.run.smithi164.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-09T12:35:16.506 INFO:teuthology.orchestra.run.smithi190.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-09T12:35:16.507 INFO:teuthology.orchestra.run.smithi190.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-09T12:35:16.511 INFO:teuthology.orchestra.run.smithi190.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-09T12:35:16.512 INFO:teuthology.orchestra.run.smithi190.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-09T12:35:16.513 INFO:teuthology.orchestra.run.smithi190.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-09T12:35:16.516 INFO:teuthology.orchestra.run.smithi190.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-09T12:35:16.517 INFO:teuthology.orchestra.run.smithi190.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-09T12:35:16.518 INFO:teuthology.orchestra.run.smithi190.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-09T12:35:16.563 INFO:teuthology.orchestra.run.smithi164.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-09T12:35:16.566 INFO:teuthology.orchestra.run.smithi164.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-09T12:35:16.584 INFO:teuthology.orchestra.run.smithi190.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-09T12:35:16.588 INFO:teuthology.orchestra.run.smithi190.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-09T12:35:16.641 INFO:teuthology.orchestra.run.smithi164.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-11-09T12:35:16.646 INFO:teuthology.orchestra.run.smithi164.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-09T12:35:16.646 INFO:teuthology.orchestra.run.smithi164.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-09T12:35:16.647 INFO:teuthology.orchestra.run.smithi164.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-09T12:35:16.648 INFO:teuthology.orchestra.run.smithi164.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-09T12:35:16.649 INFO:teuthology.orchestra.run.smithi164.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-09T12:35:16.652 INFO:teuthology.orchestra.run.smithi164.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-09T12:35:16.652 INFO:teuthology.orchestra.run.smithi164.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-09T12:35:16.662 INFO:teuthology.orchestra.run.smithi190.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-11-09T12:35:16.668 INFO:teuthology.orchestra.run.smithi190.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-09T12:35:16.668 INFO:teuthology.orchestra.run.smithi190.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-09T12:35:16.669 INFO:teuthology.orchestra.run.smithi190.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-09T12:35:16.669 INFO:teuthology.orchestra.run.smithi190.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-09T12:35:16.671 INFO:teuthology.orchestra.run.smithi190.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-09T12:35:16.673 INFO:teuthology.orchestra.run.smithi190.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-09T12:35:16.674 INFO:teuthology.orchestra.run.smithi190.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-09T12:35:16.719 INFO:teuthology.orchestra.run.smithi164.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-09T12:35:16.720 INFO:teuthology.orchestra.run.smithi164.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-09T12:35:16.740 INFO:teuthology.orchestra.run.smithi190.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-09T12:35:16.741 INFO:teuthology.orchestra.run.smithi190.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-09T12:35:16.797 INFO:teuthology.orchestra.run.smithi164.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-09T12:35:16.818 INFO:teuthology.orchestra.run.smithi164.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-09T12:35:16.819 INFO:teuthology.orchestra.run.smithi190.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-09T12:35:16.819 INFO:teuthology.orchestra.run.smithi164.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-09T12:35:16.820 INFO:teuthology.orchestra.run.smithi164.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-09T12:35:16.839 INFO:teuthology.orchestra.run.smithi190.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-09T12:35:16.840 INFO:teuthology.orchestra.run.smithi190.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-09T12:35:16.840 INFO:teuthology.orchestra.run.smithi190.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-09T12:35:16.926 INFO:teuthology.orchestra.run.smithi190.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-09T12:35:16.955 INFO:teuthology.orchestra.run.smithi164.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-09T12:35:16.955 INFO:teuthology.orchestra.run.smithi164.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-09T12:35:16.957 INFO:teuthology.orchestra.run.smithi164.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-09T12:35:16.966 INFO:teuthology.orchestra.run.smithi164.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-09T12:35:16.966 INFO:teuthology.orchestra.run.smithi164.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-09T12:35:16.967 INFO:teuthology.orchestra.run.smithi164.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-09T12:35:16.968 INFO:teuthology.orchestra.run.smithi164.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-09T12:35:16.969 INFO:teuthology.orchestra.run.smithi164.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-09T12:35:16.970 INFO:teuthology.orchestra.run.smithi164.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-09T12:35:16.971 INFO:teuthology.orchestra.run.smithi164.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-7008-gafece648-1bionic [5,569 kB] 2020-11-09T12:35:16.976 INFO:teuthology.orchestra.run.smithi190.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-09T12:35:16.976 INFO:teuthology.orchestra.run.smithi190.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-09T12:35:16.986 INFO:teuthology.orchestra.run.smithi190.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-09T12:35:16.986 INFO:teuthology.orchestra.run.smithi190.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-09T12:35:16.987 INFO:teuthology.orchestra.run.smithi190.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-09T12:35:16.988 INFO:teuthology.orchestra.run.smithi190.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-09T12:35:16.990 INFO:teuthology.orchestra.run.smithi190.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-09T12:35:16.990 INFO:teuthology.orchestra.run.smithi190.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-09T12:35:17.263 INFO:teuthology.orchestra.run.smithi190.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-7008-gafece648-1bionic [5,569 kB] 2020-11-09T12:35:17.305 INFO:teuthology.orchestra.run.smithi164.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-7008-gafece648-1bionic [165 kB] 2020-11-09T12:35:17.337 INFO:teuthology.orchestra.run.smithi164.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-7008-gafece648-1bionic [1,360 kB] 2020-11-09T12:35:17.412 INFO:teuthology.orchestra.run.smithi164.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-7008-gafece648-1bionic [6,653 kB] 2020-11-09T12:35:17.654 INFO:teuthology.orchestra.run.smithi190.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-7008-gafece648-1bionic [165 kB] 2020-11-09T12:35:17.659 INFO:teuthology.orchestra.run.smithi190.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-7008-gafece648-1bionic [1,360 kB] 2020-11-09T12:35:17.777 INFO:teuthology.orchestra.run.smithi190.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-7008-gafece648-1bionic [6,653 kB] 2020-11-09T12:35:17.819 INFO:teuthology.orchestra.run.smithi164.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-7008-gafece648-1bionic [24.6 MB] 2020-11-09T12:35:18.219 INFO:teuthology.orchestra.run.smithi190.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-7008-gafece648-1bionic [24.6 MB] 2020-11-09T12:35:19.349 INFO:teuthology.orchestra.run.smithi164.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-7008-gafece648-1bionic [3,804 B] 2020-11-09T12:35:19.349 INFO:teuthology.orchestra.run.smithi164.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-7008-gafece648-1bionic [562 kB] 2020-11-09T12:35:19.365 INFO:teuthology.orchestra.run.smithi164.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-7008-gafece648-1bionic [1,942 kB] 2020-11-09T12:35:19.501 INFO:teuthology.orchestra.run.smithi164.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-7008-gafece648-1bionic [51.5 kB] 2020-11-09T12:35:19.502 INFO:teuthology.orchestra.run.smithi164.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-7008-gafece648-1bionic [55.9 kB] 2020-11-09T12:35:19.504 INFO:teuthology.orchestra.run.smithi164.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-7008-gafece648-1bionic [2,875 kB] 2020-11-09T12:35:19.672 INFO:teuthology.orchestra.run.smithi164.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-7008-gafece648-1bionic [7,883 kB] 2020-11-09T12:35:19.953 INFO:teuthology.orchestra.run.smithi190.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-7008-gafece648-1bionic [3,804 B] 2020-11-09T12:35:19.954 INFO:teuthology.orchestra.run.smithi190.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-7008-gafece648-1bionic [562 kB] 2020-11-09T12:35:20.006 INFO:teuthology.orchestra.run.smithi190.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-7008-gafece648-1bionic [1,942 kB] 2020-11-09T12:35:20.118 INFO:teuthology.orchestra.run.smithi190.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-7008-gafece648-1bionic [51.5 kB] 2020-11-09T12:35:20.120 INFO:teuthology.orchestra.run.smithi190.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-7008-gafece648-1bionic [55.9 kB] 2020-11-09T12:35:20.122 INFO:teuthology.orchestra.run.smithi190.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-7008-gafece648-1bionic [2,875 kB] 2020-11-09T12:35:20.155 INFO:teuthology.orchestra.run.smithi164.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-7008-gafece648-1bionic [3,924 B] 2020-11-09T12:35:20.155 INFO:teuthology.orchestra.run.smithi164.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-7008-gafece648-1bionic [22.3 kB] 2020-11-09T12:35:20.156 INFO:teuthology.orchestra.run.smithi164.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-7008-gafece648-1bionic [49.6 MB] 2020-11-09T12:35:20.346 INFO:teuthology.orchestra.run.smithi190.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-7008-gafece648-1bionic [7,883 kB] 2020-11-09T12:35:20.910 INFO:teuthology.orchestra.run.smithi190.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-7008-gafece648-1bionic [3,924 B] 2020-11-09T12:35:20.910 INFO:teuthology.orchestra.run.smithi190.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-7008-gafece648-1bionic [22.3 kB] 2020-11-09T12:35:20.910 INFO:teuthology.orchestra.run.smithi190.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-7008-gafece648-1bionic [49.6 MB] 2020-11-09T12:35:22.936 INFO:teuthology.orchestra.run.smithi164.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-7008-gafece648-1bionic [19.5 kB] 2020-11-09T12:35:22.937 INFO:teuthology.orchestra.run.smithi164.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-7008-gafece648-1bionic [9,833 kB] 2020-11-09T12:35:23.470 INFO:teuthology.orchestra.run.smithi164.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-7008-gafece648-1bionic [76.8 kB] 2020-11-09T12:35:24.090 INFO:teuthology.orchestra.run.smithi190.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-7008-gafece648-1bionic [19.5 kB] 2020-11-09T12:35:24.090 INFO:teuthology.orchestra.run.smithi190.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-7008-gafece648-1bionic [9,833 kB] 2020-11-09T12:35:24.154 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 168 MB in 9s (19.5 MB/s) 2020-11-09T12:35:24.301 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-09T12:35:24.341 INFO:teuthology.orchestra.run.smithi164.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102071 files and directories currently installed.) 2020-11-09T12:35:24.344 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-09T12:35:24.374 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-09T12:35:24.409 INFO:teuthology.orchestra.run.smithi190.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-ofriedma-prefetch-range-i21041/afece648f8ccac358b59a7b305a02d8c55dc908d/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-7008-gafece648-1bionic [76.8 kB] 2020-11-09T12:35:24.694 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-09T12:35:24.706 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-09T12:35:24.786 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-09T12:35:25.104 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../002-librbd1_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:25.108 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 168 MB in 10s (17.6 MB/s) 2020-11-09T12:35:25.212 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librbd1 (16.0.0-7008-gafece648-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-09T12:35:25.269 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-09T12:35:25.305 INFO:teuthology.orchestra.run.smithi190.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102071 files and directories currently installed.) 2020-11-09T12:35:25.308 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-09T12:35:25.351 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-09T12:35:25.692 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-09T12:35:25.705 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-09T12:35:25.752 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../003-librados2_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:25.793 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-09T12:35:25.879 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librados2 (16.0.0-7008-gafece648-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-09T12:35:26.120 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../002-librbd1_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:26.241 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librbd1 (16.0.0-7008-gafece648-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-09T12:35:26.540 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libcephfs2. 2020-11-09T12:35:26.552 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:26.590 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libcephfs2 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:26.780 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../003-librados2_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:26.886 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librados2 (16.0.0-7008-gafece648-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-11-09T12:35:26.906 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rados. 2020-11-09T12:35:26.919 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../005-python3-rados_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:26.965 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rados (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:27.324 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-09T12:35:27.336 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:35:27.374 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-ceph-argparse (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:27.513 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libcephfs2. 2020-11-09T12:35:27.525 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:27.567 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libcephfs2 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:27.643 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cephfs. 2020-11-09T12:35:27.655 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:27.685 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cephfs (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:27.838 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rados. 2020-11-09T12:35:27.851 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../005-python3-rados_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:27.880 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rados (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:28.029 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-09T12:35:28.041 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:35:28.079 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-ceph-common (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:28.181 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-09T12:35:28.193 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:35:28.231 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-ceph-argparse (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:28.374 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-prettytable. 2020-11-09T12:35:28.387 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-11-09T12:35:28.424 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-09T12:35:28.516 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cephfs. 2020-11-09T12:35:28.528 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:28.567 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cephfs (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:28.676 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rbd. 2020-11-09T12:35:28.689 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:28.726 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rbd (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:28.903 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-09T12:35:28.915 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:35:28.953 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-ceph-common (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:29.037 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-09T12:35:29.050 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-09T12:35:29.079 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-09T12:35:29.222 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-prettytable. 2020-11-09T12:35:29.234 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-11-09T12:35:29.272 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-09T12:35:29.406 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-09T12:35:29.417 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-09T12:35:29.454 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-09T12:35:29.483 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rbd. 2020-11-09T12:35:29.496 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:29.524 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rbd (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:29.790 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librgw2. 2020-11-09T12:35:29.802 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../013-librgw2_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:29.840 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librgw2 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:29.885 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-09T12:35:29.898 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-09T12:35:29.935 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-09T12:35:30.273 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-09T12:35:30.285 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-09T12:35:30.323 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-09T12:35:30.376 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rgw. 2020-11-09T12:35:30.388 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:30.427 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rgw (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:30.600 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librgw2. 2020-11-09T12:35:30.611 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../013-librgw2_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:30.642 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librgw2 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:30.745 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-09T12:35:30.756 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-09T12:35:30.795 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-09T12:35:31.039 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-09T12:35:31.051 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-09T12:35:31.089 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-09T12:35:31.186 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rgw. 2020-11-09T12:35:31.198 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:31.236 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rgw (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:31.459 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package liboath0. 2020-11-09T12:35:31.470 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-11-09T12:35:31.509 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-09T12:35:31.538 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-09T12:35:31.551 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-09T12:35:31.605 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-09T12:35:31.833 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libradosstriper1. 2020-11-09T12:35:31.843 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:31.849 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-09T12:35:31.862 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-09T12:35:31.877 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libradosstriper1 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:31.891 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-09T12:35:32.261 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liboath0. 2020-11-09T12:35:32.273 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-11-09T12:35:32.311 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-09T12:35:32.325 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-common. 2020-11-09T12:35:32.337 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../019-ceph-common_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:32.376 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-common (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:32.592 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libradosstriper1. 2020-11-09T12:35:32.604 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:32.647 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libradosstriper1 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:33.110 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-common. 2020-11-09T12:35:33.121 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../019-ceph-common_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:33.152 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-common (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:34.286 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-base. 2020-11-09T12:35:34.297 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../020-ceph-base_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:34.346 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-base (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:35.059 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-base. 2020-11-09T12:35:35.072 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../020-ceph-base_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:35.111 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-base (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:35.268 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-dateutil. 2020-11-09T12:35:35.280 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-11-09T12:35:35.321 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-09T12:35:35.646 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-09T12:35:35.659 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-09T12:35:35.696 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-09T12:35:35.915 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-09T12:35:35.928 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-09T12:35:35.957 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-09T12:35:35.987 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-dateutil. 2020-11-09T12:35:35.997 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-11-09T12:35:36.046 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-09T12:35:36.260 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cryptography. 2020-11-09T12:35:36.272 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-09T12:35:36.326 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-09T12:35:36.352 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-09T12:35:36.364 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-09T12:35:36.411 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-09T12:35:36.680 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-09T12:35:36.693 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-09T12:35:36.701 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-openssl. 2020-11-09T12:35:36.713 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-09T12:35:36.730 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-09T12:35:36.751 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-09T12:35:37.000 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-09T12:35:37.012 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:35:37.049 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cryptography. 2020-11-09T12:35:37.050 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-modules-core (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:37.060 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-09T12:35:37.099 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-09T12:35:37.396 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-09T12:35:37.409 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-09T12:35:37.446 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-09T12:35:37.448 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-openssl. 2020-11-09T12:35:37.459 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-09T12:35:37.515 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-09T12:35:37.748 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-09T12:35:37.758 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-09T12:35:37.759 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:35:37.769 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-09T12:35:37.789 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-modules-core (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:37.807 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-09T12:35:38.411 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jwt. 2020-11-09T12:35:38.422 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-09T12:35:38.461 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-09T12:35:38.485 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-09T12:35:38.498 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-09T12:35:38.535 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-09T12:35:38.730 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-logutils. 2020-11-09T12:35:38.742 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-11-09T12:35:38.772 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-09T12:35:38.804 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-09T12:35:38.817 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-09T12:35:38.846 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-09T12:35:39.083 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-09T12:35:39.096 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-09T12:35:39.133 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-09T12:35:39.157 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jwt. 2020-11-09T12:35:39.171 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-09T12:35:39.208 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-09T12:35:39.435 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-mako. 2020-11-09T12:35:39.448 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-09T12:35:39.469 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-logutils. 2020-11-09T12:35:39.481 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-11-09T12:35:39.485 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-09T12:35:39.518 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-09T12:35:39.813 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-09T12:35:39.821 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-09T12:35:39.825 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-09T12:35:39.834 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-09T12:35:39.863 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-09T12:35:39.871 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-09T12:35:40.141 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-09T12:35:40.152 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-09T12:35:40.174 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-mako. 2020-11-09T12:35:40.186 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-09T12:35:40.191 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-09T12:35:40.224 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-09T12:35:40.466 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-webob. 2020-11-09T12:35:40.478 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-09T12:35:40.510 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-09T12:35:40.520 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-09T12:35:40.523 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-09T12:35:40.552 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-09T12:35:40.834 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-bs4. 2020-11-09T12:35:40.845 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-11-09T12:35:40.846 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-09T12:35:40.858 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-09T12:35:40.884 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-09T12:35:40.896 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-09T12:35:41.209 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-webob. 2020-11-09T12:35:41.220 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-waitress. 2020-11-09T12:35:41.221 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-09T12:35:41.232 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-11-09T12:35:41.259 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-09T12:35:41.271 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-09T12:35:41.561 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-bs4. 2020-11-09T12:35:41.572 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-11-09T12:35:41.581 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-tempita. 2020-11-09T12:35:41.592 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-11-09T12:35:41.611 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-09T12:35:41.630 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-09T12:35:41.927 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-paste. 2020-11-09T12:35:41.940 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-09T12:35:41.940 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-waitress. 2020-11-09T12:35:41.952 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-11-09T12:35:41.976 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-09T12:35:41.993 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-09T12:35:42.282 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-09T12:35:42.292 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-09T12:35:42.301 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-tempita. 2020-11-09T12:35:42.314 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-11-09T12:35:42.330 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-09T12:35:42.351 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-09T12:35:42.645 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-09T12:35:42.656 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-09T12:35:42.662 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-paste. 2020-11-09T12:35:42.674 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-09T12:35:42.693 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-09T12:35:42.714 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-09T12:35:42.993 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-webtest. 2020-11-09T12:35:43.005 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-09T12:35:43.035 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-09T12:35:43.043 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-09T12:35:43.047 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-09T12:35:43.086 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-09T12:35:43.425 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-09T12:35:43.436 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-09T12:35:43.485 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-09T12:35:43.498 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pecan. 2020-11-09T12:35:43.510 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-11-09T12:35:43.548 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-09T12:35:43.798 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-webtest. 2020-11-09T12:35:43.809 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-09T12:35:43.849 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-09T12:35:43.923 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-09T12:35:43.936 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-09T12:35:44.002 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-09T12:35:44.203 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pecan. 2020-11-09T12:35:44.215 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-11-09T12:35:44.253 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-09T12:35:44.318 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr. 2020-11-09T12:35:44.331 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:44.371 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:44.602 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-09T12:35:44.615 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-09T12:35:44.644 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-09T12:35:44.806 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mon. 2020-11-09T12:35:44.816 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:44.857 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mon (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:44.981 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr. 2020-11-09T12:35:44.993 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:45.024 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:45.488 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mon. 2020-11-09T12:35:45.500 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:45.539 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mon (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:45.821 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-osd. 2020-11-09T12:35:45.832 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:45.871 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-osd (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:46.514 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-osd. 2020-11-09T12:35:46.526 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:46.557 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-osd (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:47.861 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph. 2020-11-09T12:35:47.871 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../048-ceph_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:47.913 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:48.253 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-fuse. 2020-11-09T12:35:48.265 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:48.555 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph. 2020-11-09T12:35:48.567 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../048-ceph_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:49.011 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-fuse (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:49.240 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:49.349 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mds. 2020-11-09T12:35:49.362 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:49.399 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mds (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:49.540 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-fuse. 2020-11-09T12:35:49.553 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:49.582 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-fuse (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:49.878 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package cephadm. 2020-11-09T12:35:49.891 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../051-cephadm_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:49.928 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking cephadm (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:49.968 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mds. 2020-11-09T12:35:49.981 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:50.018 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mds (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:50.217 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jinja2. 2020-11-09T12:35:50.228 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-09T12:35:50.258 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-09T12:35:50.451 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package cephadm. 2020-11-09T12:35:50.462 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../051-cephadm_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:35:50.497 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking cephadm (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:50.585 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-09T12:35:50.597 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:35:50.636 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-cephadm (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:50.821 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jinja2. 2020-11-09T12:35:50.834 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-09T12:35:50.889 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-09T12:35:50.959 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-09T12:35:50.971 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-11-09T12:35:51.010 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-09T12:35:51.177 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-09T12:35:51.189 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:35:51.231 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-cephadm (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:51.298 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-routes. 2020-11-09T12:35:51.311 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-11-09T12:35:51.349 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-09T12:35:51.553 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-09T12:35:51.565 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-11-09T12:35:51.594 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-09T12:35:51.671 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-09T12:35:51.684 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:35:51.721 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-dashboard (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:51.872 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-routes. 2020-11-09T12:35:51.886 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-11-09T12:35:51.931 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-09T12:35:52.234 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-09T12:35:52.245 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:35:52.284 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-dashboard (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:52.487 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-numpy. 2020-11-09T12:35:52.502 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-09T12:35:52.545 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-09T12:35:53.100 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-numpy. 2020-11-09T12:35:53.113 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-09T12:35:53.141 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-09T12:35:53.202 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-decorator. 2020-11-09T12:35:53.215 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-11-09T12:35:53.260 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-09T12:35:53.541 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-scipy. 2020-11-09T12:35:53.554 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-09T12:35:53.599 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-09T12:35:53.749 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-decorator. 2020-11-09T12:35:53.762 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-11-09T12:35:53.817 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-09T12:35:54.138 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-scipy. 2020-11-09T12:35:54.150 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-09T12:35:54.188 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-09T12:35:54.802 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-09T12:35:54.813 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-09T12:35:54.852 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-09T12:35:55.320 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-joblib. 2020-11-09T12:35:55.334 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-11-09T12:35:55.378 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-09T12:35:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-09T12:35:55.409 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-09T12:35:55.461 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-09T12:35:55.707 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-sklearn. 2020-11-09T12:35:55.718 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-11-09T12:35:55.757 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-09T12:35:55.906 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-joblib. 2020-11-09T12:35:55.919 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-11-09T12:35:55.956 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-09T12:35:56.271 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-sklearn. 2020-11-09T12:35:56.283 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-11-09T12:35:56.314 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-09T12:35:56.366 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-09T12:35:56.379 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:35:56.427 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:56.919 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-09T12:35:56.932 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:35:56.977 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:35:57.562 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-monotonic. 2020-11-09T12:35:57.575 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-11-09T12:35:57.621 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-09T12:35:57.945 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-fasteners. 2020-11-09T12:35:57.957 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-11-09T12:35:57.995 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-09T12:35:58.033 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-monotonic. 2020-11-09T12:35:58.044 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-11-09T12:35:58.084 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-09T12:35:58.317 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-httplib2. 2020-11-09T12:35:58.331 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-09T12:35:58.376 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-09T12:35:58.401 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-fasteners. 2020-11-09T12:35:58.414 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-11-09T12:35:58.451 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-09T12:35:58.713 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-09T12:35:58.725 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-09T12:35:58.756 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-httplib2. 2020-11-09T12:35:58.763 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-09T12:35:58.769 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-09T12:35:58.808 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-09T12:35:59.114 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-09T12:35:59.126 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-09T12:35:59.150 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-09T12:35:59.161 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-09T12:35:59.165 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-09T12:35:59.193 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-09T12:35:59.473 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rsa. 2020-11-09T12:35:59.486 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-11-09T12:35:59.496 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-09T12:35:59.507 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-09T12:35:59.532 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-09T12:35:59.547 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-09T12:35:59.844 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-09T12:35:59.858 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-09T12:35:59.873 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rsa. 2020-11-09T12:35:59.884 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-11-09T12:35:59.894 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-09T12:35:59.923 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-09T12:36:00.244 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-09T12:36:00.255 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-09T12:36:00.295 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-09T12:36:00.352 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-09T12:36:00.363 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-09T12:36:00.437 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-09T12:36:00.740 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-09T12:36:00.754 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-09T12:36:00.768 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-setuptools. 2020-11-09T12:36:00.781 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-11-09T12:36:00.799 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-09T12:36:00.813 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-09T12:36:01.144 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-setuptools. 2020-11-09T12:36:01.157 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-11-09T12:36:01.211 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-09T12:36:01.218 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-websocket. 2020-11-09T12:36:01.230 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-09T12:36:01.270 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-09T12:36:01.595 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-09T12:36:01.606 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-09T12:36:01.608 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-websocket. 2020-11-09T12:36:01.620 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-09T12:36:01.648 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-09T12:36:01.667 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-09T12:36:01.986 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-09T12:36:01.998 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-09T12:36:02.036 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-09T12:36:02.081 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-09T12:36:02.095 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:36:02.139 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:02.432 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-09T12:36:02.449 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:36:02.470 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-09T12:36:02.482 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-11-09T12:36:02.491 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:02.522 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-09T12:36:02.860 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-09T12:36:02.870 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-09T12:36:02.871 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-11-09T12:36:02.882 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-09T12:36:02.910 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-09T12:36:02.923 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-09T12:36:03.221 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-09T12:36:03.233 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:36:03.259 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-09T12:36:03.270 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-09T12:36:03.278 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-rook (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:03.318 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-09T12:36:03.617 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-09T12:36:03.629 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-11-09T12:36:03.642 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-09T12:36:03.656 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-7008-gafece648-1bionic_all.deb ... 2020-11-09T12:36:03.668 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-09T12:36:03.695 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-rook (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:03.991 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-09T12:36:04.004 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-09T12:36:04.063 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-09T12:36:04.075 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-11-09T12:36:04.120 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-09T12:36:04.132 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-09T12:36:04.387 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package jq. 2020-11-09T12:36:04.401 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-11-09T12:36:04.437 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-09T12:36:04.460 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-09T12:36:04.474 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-09T12:36:04.588 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-09T12:36:04.715 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package socat. 2020-11-09T12:36:04.727 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-09T12:36:04.766 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-09T12:36:04.865 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package jq. 2020-11-09T12:36:04.878 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-11-09T12:36:04.915 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-09T12:36:05.151 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package socat. 2020-11-09T12:36:05.152 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-09T12:36:05.163 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-09T12:36:05.164 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-09T12:36:05.193 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-09T12:36:05.194 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-09T12:36:05.455 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package xmlstarlet. 2020-11-09T12:36:05.468 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-09T12:36:05.505 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-09T12:36:05.613 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-09T12:36:05.624 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-09T12:36:05.663 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-09T12:36:05.836 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-test. 2020-11-09T12:36:05.850 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../085-ceph-test_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:36:05.894 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-test (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:05.949 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package xmlstarlet. 2020-11-09T12:36:05.962 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-09T12:36:05.999 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-09T12:36:06.364 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-test. 2020-11-09T12:36:06.376 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../085-ceph-test_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:36:06.413 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-test (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:09.535 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-09T12:36:09.546 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:36:09.585 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libcephfs-dev (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:09.832 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package nvme-cli. 2020-11-09T12:36:09.845 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-11-09T12:36:09.883 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-11-09T12:36:10.097 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-09T12:36:10.110 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:36:10.148 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libcephfs-dev (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:10.248 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-attr. 2020-11-09T12:36:10.261 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-11-09T12:36:10.299 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-09T12:36:10.395 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package nvme-cli. 2020-11-09T12:36:10.408 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-11-09T12:36:10.454 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-11-09T12:36:10.589 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-funcsigs. 2020-11-09T12:36:10.604 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-11-09T12:36:10.648 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-09T12:36:10.803 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-attr. 2020-11-09T12:36:10.815 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-11-09T12:36:10.845 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-09T12:36:10.972 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-pluggy. 2020-11-09T12:36:10.985 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-11-09T12:36:11.023 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-09T12:36:11.152 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-funcsigs. 2020-11-09T12:36:11.164 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-11-09T12:36:11.203 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-09T12:36:11.313 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-py. 2020-11-09T12:36:11.326 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-11-09T12:36:11.371 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-09T12:36:11.537 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-pluggy. 2020-11-09T12:36:11.547 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-11-09T12:36:11.585 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-09T12:36:11.683 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-pytest. 2020-11-09T12:36:11.696 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-11-09T12:36:11.734 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-09T12:36:11.875 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-py. 2020-11-09T12:36:11.889 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-11-09T12:36:11.935 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-09T12:36:12.068 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-attr. 2020-11-09T12:36:12.080 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-11-09T12:36:12.129 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-09T12:36:12.270 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-pytest. 2020-11-09T12:36:12.282 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-11-09T12:36:12.320 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-09T12:36:12.455 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cycler. 2020-11-09T12:36:12.468 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-11-09T12:36:12.506 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-09T12:36:12.674 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-attr. 2020-11-09T12:36:12.684 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-11-09T12:36:12.723 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-09T12:36:12.813 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-webencodings. 2020-11-09T12:36:12.826 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-11-09T12:36:12.859 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-09T12:36:13.048 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cycler. 2020-11-09T12:36:13.062 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-11-09T12:36:13.099 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-09T12:36:13.165 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-html5lib. 2020-11-09T12:36:13.178 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-11-09T12:36:13.215 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-09T12:36:13.393 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-webencodings. 2020-11-09T12:36:13.406 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-11-09T12:36:13.444 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-09T12:36:13.542 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-09T12:36:13.555 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-09T12:36:13.600 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-09T12:36:13.749 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-html5lib. 2020-11-09T12:36:13.760 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-11-09T12:36:13.802 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-09T12:36:14.027 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-09T12:36:14.041 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-09T12:36:14.085 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-09T12:36:14.134 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-09T12:36:14.148 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-09T12:36:14.184 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-09T12:36:14.396 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-tz. 2020-11-09T12:36:14.409 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-11-09T12:36:14.446 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-09T12:36:14.659 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-09T12:36:14.672 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-09T12:36:14.717 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-09T12:36:14.740 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-09T12:36:14.754 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-09T12:36:14.798 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-09T12:36:15.045 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-tz. 2020-11-09T12:36:15.058 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-11-09T12:36:15.095 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-09T12:36:15.365 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-09T12:36:15.377 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-09T12:36:15.406 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-09T12:36:15.456 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-olefile. 2020-11-09T12:36:15.467 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-11-09T12:36:15.506 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-09T12:36:15.809 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pastescript. 2020-11-09T12:36:15.823 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-11-09T12:36:15.877 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-09T12:36:16.097 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-olefile. 2020-11-09T12:36:16.111 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-11-09T12:36:16.172 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-09T12:36:16.221 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-09T12:36:16.233 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-09T12:36:16.270 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-09T12:36:16.491 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pastescript. 2020-11-09T12:36:16.505 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-11-09T12:36:16.549 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-09T12:36:16.676 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pluggy. 2020-11-09T12:36:16.687 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-11-09T12:36:16.734 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-09T12:36:16.903 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-09T12:36:16.916 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-09T12:36:16.953 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-09T12:36:17.020 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-py. 2020-11-09T12:36:17.034 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-11-09T12:36:17.078 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-09T12:36:17.308 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pluggy. 2020-11-09T12:36:17.322 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-11-09T12:36:17.383 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-09T12:36:17.406 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-09T12:36:17.421 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-09T12:36:17.469 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-09T12:36:17.628 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-py. 2020-11-09T12:36:17.639 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-11-09T12:36:17.669 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-09T12:36:17.776 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pytest. 2020-11-09T12:36:17.787 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-11-09T12:36:17.826 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-09T12:36:17.997 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-09T12:36:18.011 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-09T12:36:18.055 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-09T12:36:18.174 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-simplejson. 2020-11-09T12:36:18.188 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-09T12:36:18.232 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-09T12:36:18.375 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pytest. 2020-11-09T12:36:18.388 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-11-09T12:36:18.425 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-09T12:36:18.551 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-09T12:36:18.565 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-09T12:36:18.610 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-09T12:36:18.714 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-simplejson. 2020-11-09T12:36:18.727 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-09T12:36:18.756 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-09T12:36:19.080 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package radosgw. 2020-11-09T12:36:19.084 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-09T12:36:19.093 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../110-radosgw_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:36:19.098 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-09T12:36:19.130 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking radosgw (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:19.142 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-09T12:36:19.604 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package radosgw. 2020-11-09T12:36:19.619 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../110-radosgw_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:36:19.662 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking radosgw (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:20.145 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package rbd-fuse. 2020-11-09T12:36:20.157 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:36:20.195 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking rbd-fuse (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:20.487 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-09T12:36:20.677 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package rbd-fuse. 2020-11-09T12:36:20.690 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-7008-gafece648-1bionic_amd64.deb ... 2020-11-09T12:36:20.727 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking rbd-fuse (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:20.766 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-09T12:36:21.020 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-09T12:36:21.126 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-09T12:36:21.252 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-09T12:36:21.332 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-09T12:36:21.683 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-09T12:36:21.821 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-09T12:36:21.857 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-09T12:36:22.208 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-09T12:36:22.398 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-09T12:36:22.502 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-09T12:36:22.786 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-09T12:36:22.791 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-09T12:36:22.987 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-09T12:36:23.075 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-09T12:36:23.113 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-09T12:36:23.362 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-09T12:36:23.456 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-09T12:36:23.563 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-09T12:36:23.689 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-09T12:36:23.810 INFO:teuthology.orchestra.run.smithi164.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-09T12:36:24.018 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-09T12:36:24.113 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-09T12:36:24.350 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-09T12:36:24.422 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-ceph-argparse (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:24.638 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-09T12:36:24.715 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-09T12:36:24.965 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-ceph-argparse (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:24.990 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-09T12:36:25.091 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-09T12:36:25.274 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-09T12:36:25.575 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-09T12:36:25.684 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-09T12:36:26.006 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-09T12:36:26.308 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-09T12:36:26.609 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-py (1.5.2-1) ... 2020-11-09T12:36:26.616 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-09T12:36:26.835 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-09T12:36:26.926 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-09T12:36:27.202 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-py (1.5.2-1) ... 2020-11-09T12:36:27.211 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-09T12:36:27.301 INFO:teuthology.orchestra.run.smithi164.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-09T12:36:27.437 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-09T12:36:27.546 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-09T12:36:27.672 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-09T12:36:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-09T12:36:27.848 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-09T12:36:27.870 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-09T12:36:28.123 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-09T12:36:28.124 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-09T12:36:28.249 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-09T12:36:28.442 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-09T12:36:28.446 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-09T12:36:28.739 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-09T12:36:28.756 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-09T12:36:29.073 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-09T12:36:29.090 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-09T12:36:29.391 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-09T12:36:29.400 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-09T12:36:29.685 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up cephadm (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:29.751 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-09T12:36:30.060 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-09T12:36:30.361 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up cephadm (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:30.393 INFO:teuthology.orchestra.run.smithi164.stdout:Adding system user cephadm....done 2020-11-09T12:36:30.543 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-09T12:36:30.827 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-09T12:36:30.953 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-11-09T12:36:31.014 INFO:teuthology.orchestra.run.smithi190.stdout:Adding system user cephadm....done 2020-11-09T12:36:31.082 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-09T12:36:31.170 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-09T12:36:31.430 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-09T12:36:31.471 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-09T12:36:31.572 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-11-09T12:36:31.673 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-09T12:36:31.723 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-09T12:36:31.824 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-09T12:36:32.008 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-09T12:36:32.142 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-ceph-common (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:32.352 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-09T12:36:32.452 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-09T12:36:32.482 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-09T12:36:32.749 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-09T12:36:32.805 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-ceph-common (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:32.875 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-09T12:36:33.098 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-09T12:36:33.209 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-09T12:36:33.374 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-09T12:36:33.500 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-09T12:36:33.560 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-09T12:36:33.650 INFO:teuthology.orchestra.run.smithi164.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-09T12:36:33.682 INFO:teuthology.orchestra.run.smithi164.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-09T12:36:33.868 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-09T12:36:33.929 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-09T12:36:34.211 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-09T12:36:34.222 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-09T12:36:34.300 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-09T12:36:34.333 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-09T12:36:34.556 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-09T12:36:34.579 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-09T12:36:34.866 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-09T12:36:34.889 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-09T12:36:35.154 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-09T12:36:35.173 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-09T12:36:35.354 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-09T12:36:35.510 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-09T12:36:35.731 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-09T12:36:35.820 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-09T12:36:35.846 INFO:teuthology.orchestra.run.smithi164.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-09T12:36:36.021 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-09T12:36:36.091 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-09T12:36:36.250 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-09T12:36:36.376 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-09T12:36:36.431 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-09T12:36:36.529 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-09T12:36:36.636 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-09T12:36:36.770 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-09T12:36:36.783 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-09T12:36:36.959 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-09T12:36:37.086 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-09T12:36:37.158 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-09T12:36:37.285 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-09T12:36:37.339 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-09T12:36:37.440 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-09T12:36:37.585 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-09T12:36:37.800 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-09T12:36:37.886 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-09T12:36:37.934 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-09T12:36:38.170 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-09T12:36:38.254 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-09T12:36:38.497 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-09T12:36:38.553 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-09T12:36:38.838 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-09T12:36:38.856 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-09T12:36:39.149 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-09T12:36:39.170 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-09T12:36:39.455 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-09T12:36:39.520 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-09T12:36:39.838 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-09T12:36:40.093 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-09T12:36:40.136 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-09T12:36:40.489 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-09T12:36:40.515 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-09T12:36:40.808 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-09T12:36:40.845 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-09T12:36:40.938 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-09T12:36:41.169 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-09T12:36:41.300 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-09T12:36:42.110 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-09T12:36:42.424 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-09T12:36:42.503 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-09T12:36:42.735 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-09T12:36:42.813 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-09T12:36:42.864 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-09T12:36:43.134 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-09T12:36:43.219 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-09T12:36:43.262 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-09T12:36:43.482 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-09T12:36:43.602 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-09T12:36:43.639 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-09T12:36:43.739 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librados2 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:43.850 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-09T12:36:43.905 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-09T12:36:44.026 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-09T12:36:44.140 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librados2 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:44.142 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libcephfs2 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:44.260 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-09T12:36:44.272 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libcephfs-dev (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:44.407 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rados (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:44.509 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-09T12:36:44.580 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libcephfs2 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:44.707 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libcephfs-dev (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:44.834 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rados (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:44.862 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-09T12:36:45.174 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-09T12:36:45.295 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-09T12:36:45.500 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-09T12:36:45.628 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-09T12:36:45.924 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-09T12:36:46.236 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-09T12:36:46.425 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-09T12:36:47.164 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-09T12:36:47.282 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-09T12:36:47.573 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libradosstriper1 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:47.704 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cephfs (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:47.993 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-09T12:36:48.009 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librgw2 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:48.134 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-fuse (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:48.306 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libradosstriper1 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:48.343 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-09T12:36:48.343 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-09T12:36:48.421 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cephfs (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:48.633 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librbd1 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:48.731 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librgw2 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:48.745 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-09T12:36:48.835 INFO:teuthology.orchestra.run.smithi164.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-09T12:36:48.843 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-fuse (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:48.859 INFO:teuthology.orchestra.run.smithi164.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-09T12:36:49.062 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-09T12:36:49.062 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-09T12:36:49.174 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-09T12:36:49.344 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librbd1 (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:49.477 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-09T12:36:49.567 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-09T12:36:49.600 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-09T12:36:49.841 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-modules-core (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:49.907 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-09T12:36:49.960 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-09T12:36:50.291 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rgw (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:50.391 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up rbd-fuse (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:50.512 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rbd (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:50.555 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-modules-core (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:50.644 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-common (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:50.666 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-09T12:36:50.985 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rgw (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:51.110 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up rbd-fuse (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:51.179 INFO:teuthology.orchestra.run.smithi164.stdout:Adding group ceph....done 2020-11-09T12:36:51.261 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rbd (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:51.387 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-common (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:51.539 INFO:teuthology.orchestra.run.smithi164.stdout:Adding system user ceph....done 2020-11-09T12:36:51.724 INFO:teuthology.orchestra.run.smithi164.stdout:Setting system user ceph properties....done 2020-11-09T12:36:51.874 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-09T12:36:51.879 INFO:teuthology.orchestra.run.smithi190.stdout:Adding group ceph....done 2020-11-09T12:36:52.083 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-09T12:36:52.300 INFO:teuthology.orchestra.run.smithi190.stdout:Adding system user ceph....done 2020-11-09T12:36:52.379 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up radosgw (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:52.491 INFO:teuthology.orchestra.run.smithi190.stdout:Setting system user ceph properties....done 2020-11-09T12:36:52.640 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-09T12:36:52.640 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-09T12:36:52.642 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-09T12:36:52.860 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-09T12:36:53.031 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-test (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:53.148 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up radosgw (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:53.157 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-base (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:53.392 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-09T12:36:53.392 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-09T12:36:53.465 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-09T12:36:53.750 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:53.776 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-test (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:53.902 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-base (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:53.955 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-09T12:36:53.956 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-09T12:36:54.226 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:54.242 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-09T12:36:54.369 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-osd (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:54.529 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:54.742 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-09T12:36:54.743 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-09T12:36:54.822 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-09T12:36:54.822 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-09T12:36:55.030 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:55.173 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-osd (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:55.297 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-cephadm (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:55.440 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-rook (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:55.574 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mds (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:55.616 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-09T12:36:55.617 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-09T12:36:55.771 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-09T12:36:55.772 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-09T12:36:55.893 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-cephadm (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:56.017 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mon (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:56.035 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-rook (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:56.178 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mds (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:56.214 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-09T12:36:56.214 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-09T12:36:56.399 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-09T12:36:56.400 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-09T12:36:56.490 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-k8sevents (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:56.615 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-dashboard (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:56.682 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mon (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:56.757 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:56.859 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-09T12:36:56.887 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-09T12:36:56.887 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-09T12:36:57.043 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-09T12:36:57.159 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-k8sevents (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:57.285 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-dashboard (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:57.419 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph (16.0.0-7008-gafece648-1bionic) ... 2020-11-09T12:36:57.546 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-09T12:36:57.738 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-09T12:36:59.078 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-09T12:36:59.113 INFO:teuthology.orchestra.run.smithi164.stdout:ureadahead will be reprofiled on next reboot 2020-11-09T12:36:59.172 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-09T12:36:59.635 INFO:teuthology.orchestra.run.smithi164.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-09T12:36:59.638 DEBUG:teuthology.parallel:result is None 2020-11-09T12:36:59.782 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-09T12:36:59.816 INFO:teuthology.orchestra.run.smithi190.stdout:ureadahead will be reprofiled on next reboot 2020-11-09T12:36:59.866 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-09T12:37:00.324 INFO:teuthology.orchestra.run.smithi190.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-09T12:37:00.326 DEBUG:teuthology.parallel:result is None 2020-11-09T12:37:00.327 INFO:teuthology.orchestra.run.smithi164:> sudo lsb_release -is 2020-11-09T12:37:00.383 INFO:teuthology.orchestra.run.smithi164.stdout:Ubuntu 2020-11-09T12:37:00.388 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-09T12:37:00.389 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=afece648f8ccac358b59a7b305a02d8c55dc908d 2020-11-09T12:37:00.564 INFO:teuthology.orchestra.run.smithi164:> dpkg-query -W -f '${Version}' ceph 2020-11-09T12:37:00.587 INFO:teuthology.orchestra.run.smithi164.stdout:16.0.0-7008-gafece648-1bionic 2020-11-09T12:37:00.588 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7008-gafece648-1bionic 2020-11-09T12:37:00.588 INFO:teuthology.task.install:The correct ceph version 16.0.0-7008-gafece648-1bionic is installed. 2020-11-09T12:37:00.589 INFO:teuthology.orchestra.run.smithi190:> sudo lsb_release -is 2020-11-09T12:37:00.646 INFO:teuthology.orchestra.run.smithi190.stdout:Ubuntu 2020-11-09T12:37:00.651 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-09T12:37:00.652 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=afece648f8ccac358b59a7b305a02d8c55dc908d 2020-11-09T12:37:00.821 INFO:teuthology.orchestra.run.smithi190:> dpkg-query -W -f '${Version}' ceph 2020-11-09T12:37:00.847 INFO:teuthology.orchestra.run.smithi190.stdout:16.0.0-7008-gafece648-1bionic 2020-11-09T12:37:00.847 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7008-gafece648-1bionic 2020-11-09T12:37:00.847 INFO:teuthology.task.install:The correct ceph version 16.0.0-7008-gafece648-1bionic is installed. 2020-11-09T12:37:00.848 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-11-09T12:37:00.848 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:00.849 INFO:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-09T12:37:00.853 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-09T12:37:00.863 INFO:teuthology.orchestra.run.smithi164.stderr:24+1 records in 2020-11-09T12:37:00.863 INFO:teuthology.orchestra.run.smithi164.stderr:24+1 records out 2020-11-09T12:37:00.863 INFO:teuthology.orchestra.run.smithi164.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000217934 s, 58.1 MB/s 2020-11-09T12:37:00.864 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:00.864 INFO:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-09T12:37:00.894 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-09T12:37:00.903 INFO:teuthology.orchestra.run.smithi190.stderr:24+1 records in 2020-11-09T12:37:00.903 INFO:teuthology.orchestra.run.smithi190.stderr:24+1 records out 2020-11-09T12:37:00.903 INFO:teuthology.orchestra.run.smithi190.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000220903 s, 57.3 MB/s 2020-11-09T12:37:00.904 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-11-09T12:37:00.905 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:00.906 INFO:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2020-11-09T12:37:00.911 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-09T12:37:00.920 INFO:teuthology.orchestra.run.smithi164.stderr:5+1 records in 2020-11-09T12:37:00.920 INFO:teuthology.orchestra.run.smithi164.stderr:5+1 records out 2020-11-09T12:37:00.920 INFO:teuthology.orchestra.run.smithi164.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.4267e-05 s, 32.4 MB/s 2020-11-09T12:37:00.921 INFO:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-09T12:37:00.975 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:00.976 INFO:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2020-11-09T12:37:00.981 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-09T12:37:00.990 INFO:teuthology.orchestra.run.smithi190.stderr:5+1 records in 2020-11-09T12:37:00.990 INFO:teuthology.orchestra.run.smithi190.stderr:5+1 records out 2020-11-09T12:37:00.990 INFO:teuthology.orchestra.run.smithi190.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.8007e-05 s, 31.2 MB/s 2020-11-09T12:37:00.991 INFO:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-09T12:37:01.048 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-11-09T12:37:01.049 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:01.049 INFO:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-09T12:37:01.054 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-09T12:37:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:299 bytes copied, 7.0154e-05 s, 4.3 MB/s 2020-11-09T12:37:01.064 INFO:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-09T12:37:01.119 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:01.120 INFO:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-09T12:37:01.124 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-09T12:37:01.133 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:01.133 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:01.133 INFO:teuthology.orchestra.run.smithi190.stderr:299 bytes copied, 6.925e-05 s, 4.3 MB/s 2020-11-09T12:37:01.134 INFO:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-09T12:37:01.191 INFO:teuthology.run_tasks:Running task ceph... 2020-11-09T12:37:01.266 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-11-09T12:37:01.267 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 777 /var/log/ceph 2020-11-09T12:37:01.269 INFO:teuthology.orchestra.run.smithi190:> sudo chmod 777 /var/log/ceph 2020-11-09T12:37:01.284 INFO:tasks.ceph:Disabling ceph logrotate... 2020-11-09T12:37:01.284 INFO:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-09T12:37:01.327 INFO:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-09T12:37:01.341 INFO:tasks.ceph:Creating extra log directories... 2020-11-09T12:37:01.342 INFO:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-09T12:37:01.379 INFO:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-09T12:37:01.412 INFO:tasks.ceph:Creating ceph cluster c1... 2020-11-09T12:37:01.413 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-09T12:37:01.414 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/ofriedma-2020-11-09_09:43:03-rgw-wip-ofriedma-prefetch-range-i21041-distro-basic-smithi/5605479', 'branch': 'wip-ofriedma-prefetch-range-i21041', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'ofriedma@redhat.com', 'first_in_suite': False, 'job_id': '5605479', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'ofriedma-2020-11-09_09:43:03-rgw-wip-ofriedma-prefetch-range-i21041-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-ofriedma-prefetch-range-i21041'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-ofriedma-prefetch-range-i21041', 'sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d'}}, 'owner': 'scheduled_ofriedma@teuthology', 'priority': 101, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-ofriedma-prefetch-range-i21041', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-ofriedma-prefetch-range-i21041/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2020-11-09_09:43:03', 'tube': 'smithi', 'user': 'ofriedma', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.11564', 'targets': {'smithi164.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDe4y9371xWFuJuIE04B1Fo1b3ZUH7GU9oATqw2xFl1QfUisADv12XzhKqc0ugtdpxyKvS/gawH3evS0UNqAISY3XCL6/tDGLyFGYGSKt64jJDB8sDZjAM70CW+2a4LHdnUOwU/1WgrN3UJt1bvxqBJx1JKxzf1tLXpbNS1ddAYdpFO8JfVRWDzAJE9A1sMDxwKLol7ugg8NcoXhgwIZy3c1njFGKsw2qLUDFlLX7uFl0ZcfXHSi/fbT75rz38IpYMnIBWsvJ7p6eeOZ2bggJEjqnxplzB53wy0tj0z++vsj1B7lAcm8poGBCEkckeoGJBs5USEpY/SJrhggEA4SUv5', 'smithi190.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQeVG89hdn6N416X1bJbSagXnM97175QAWzlHiMK4psAkbs3KVf9m0VKtccgsoREVvt7MGw3QTez9R2g4MBvQuto2PHUqUZqq5V4Pm6tBV4JnivVwTLThK/9yXXQDKyHaDV5tQ0q0KO9QtVod93+J+ksEm+zhq8BJM4l/gn08hIM/1adp4g8wOP1AjtZ+CdHm7PGUdDKTmrYpNx5Jw9dyGMKrTAcnq9sgWO7i1YlbXRIj1wTj3N6+X2DmoiI7jYLixC7D5IaSvwZ4BrnWqAKeIoG3Cbp2aQK01TLELHgAZnc9N0J/xD8rwZ2oLZTc8dXgzTkXqzStjS+v0erO1zVb5'}} 2020-11-09T12:37:01.415 INFO:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-11-09T12:37:01.439 INFO:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-11-09T12:37:01.447 INFO:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2020-11-09T12:37:01.487 INFO:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2020-11-09T12:37:01.502 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:01.503 INFO:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2020-11-09T12:37:01.545 INFO:teuthology.orchestra.run.smithi164.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-09T12:37:01.546 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/vg_nvme/lv_1 2020-11-09T12:37:01.546 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/vg_nvme/lv_2 2020-11-09T12:37:01.547 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/vg_nvme/lv_3 2020-11-09T12:37:01.547 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/vg_nvme/lv_4 2020-11-09T12:37:01.547 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:01.547 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:01.547 INFO:teuthology.orchestra.run.smithi164.stderr:72 bytes copied, 0.00015416 s, 467 kB/s 2020-11-09T12:37:01.547 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-09T12:37:01.548 INFO:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-09T12:37:01.596 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-09T12:37:01.596 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T12:37:01.597 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-11-09T12:37:01.597 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T12:37:01.597 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2020-11-09 12:34:42.065345733 +0000 2020-11-09T12:37:01.597 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2020-11-09 12:34:41.953289736 +0000 2020-11-09T12:37:01.598 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2020-11-09 12:34:41.953289736 +0000 2020-11-09T12:37:01.598 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2020-11-09T12:37:01.605 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2020-11-09T12:37:01.606 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2020-11-09T12:37:01.606 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.00040627 s, 1.3 MB/s 2020-11-09T12:37:01.610 INFO:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-09T12:37:01.659 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-09T12:37:01.660 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T12:37:01.660 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-11-09T12:37:01.660 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T12:37:01.660 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2020-11-09 12:34:42.321473727 +0000 2020-11-09T12:37:01.660 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2020-11-09 12:34:42.213419730 +0000 2020-11-09T12:37:01.661 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2020-11-09 12:34:42.213419730 +0000 2020-11-09T12:37:01.661 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2020-11-09T12:37:01.669 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2020-11-09T12:37:01.669 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2020-11-09T12:37:01.669 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.00033975 s, 1.5 MB/s 2020-11-09T12:37:01.673 INFO:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-09T12:37:01.723 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-09T12:37:01.724 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T12:37:01.724 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-11-09T12:37:01.724 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T12:37:01.725 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2020-11-09 12:34:42.557591723 +0000 2020-11-09T12:37:01.725 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2020-11-09 12:34:42.445535726 +0000 2020-11-09T12:37:01.725 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2020-11-09 12:34:42.445535726 +0000 2020-11-09T12:37:01.725 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2020-11-09T12:37:01.733 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2020-11-09T12:37:01.733 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2020-11-09T12:37:01.733 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000388667 s, 1.3 MB/s 2020-11-09T12:37:01.738 INFO:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-09T12:37:01.788 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-09T12:37:01.788 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T12:37:01.789 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-11-09T12:37:01.789 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T12:37:01.789 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2020-11-09 12:34:42.793709718 +0000 2020-11-09T12:37:01.789 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2020-11-09 12:34:42.681653720 +0000 2020-11-09T12:37:01.789 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2020-11-09 12:34:42.681653720 +0000 2020-11-09T12:37:01.790 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2020-11-09T12:37:01.797 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2020-11-09T12:37:01.798 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2020-11-09T12:37:01.798 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000411387 s, 1.2 MB/s 2020-11-09T12:37:01.802 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T12:37:01.802 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-09T12:37:01.802 INFO:tasks.ceph:Generating config... 2020-11-09T12:37:01.804 INFO:tasks.ceph:[client] debug rgw = 20 2020-11-09T12:37:01.804 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-11-09T12:37:01.804 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-11-09T12:37:01.804 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-09T12:37:01.805 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-11-09T12:37:01.805 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-11-09T12:37:01.805 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-11-09T12:37:01.805 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-11-09T12:37:01.805 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-11-09T12:37:01.806 INFO:tasks.ceph:[client] setgroup = ceph 2020-11-09T12:37:01.806 INFO:tasks.ceph:[client] setuser = ceph 2020-11-09T12:37:01.806 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-11-09T12:37:01.806 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-09T12:37:01.807 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-09T12:37:01.807 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-09T12:37:01.807 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-09T12:37:01.807 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-09T12:37:01.807 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-11-09T12:37:01.808 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-09T12:37:01.808 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-09T12:37:01.808 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-09T12:37:01.808 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-09T12:37:01.808 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-11-09T12:37:01.809 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-11-09T12:37:01.809 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-11-09T12:37:01.809 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-11-09T12:37:01.809 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-11-09T12:37:01.810 INFO:tasks.ceph:Setting up c1.mon.a... 2020-11-09T12:37:01.810 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-11-09T12:37:01.876 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/c1.keyring 2020-11-09T12:37:01.878 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-11-09T12:37:01.955 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-09T12:37:02.010 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.164')] 2020-11-09T12:37:02.010 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon host': '172.21.15.164', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-11-09T12:37:02.011 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:02.012 INFO:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-09T12:37:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-09T12:37:02.059 INFO:teuthology.orchestra.run.smithi164.stderr:6+1 records in 2020-11-09T12:37:02.059 INFO:teuthology.orchestra.run.smithi164.stderr:6+1 records out 2020-11-09T12:37:02.059 INFO:teuthology.orchestra.run.smithi164.stderr:3101 bytes (3.1 kB, 3.0 KiB) copied, 0.00011502 s, 27.0 MB/s 2020-11-09T12:37:02.060 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.164 --print /home/ubuntu/cephtest/c1.monmap 2020-11-09T12:37:02.132 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:37:02.133 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:37:02.133 INFO:teuthology.orchestra.run.smithi164.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-11-09T12:37:02.133 INFO:teuthology.orchestra.run.smithi164.stdout:monmaptool: generated fsid 77b52c3d-49b9-4299-b194-b448006caee1 2020-11-09T12:37:02.134 INFO:teuthology.orchestra.run.smithi164.stdout:epoch 0 2020-11-09T12:37:02.134 INFO:teuthology.orchestra.run.smithi164.stdout:fsid 77b52c3d-49b9-4299-b194-b448006caee1 2020-11-09T12:37:02.134 INFO:teuthology.orchestra.run.smithi164.stdout:last_changed 2020-11-09T12:37:02.135674+0000 2020-11-09T12:37:02.134 INFO:teuthology.orchestra.run.smithi164.stdout:created 2020-11-09T12:37:02.135674+0000 2020-11-09T12:37:02.134 INFO:teuthology.orchestra.run.smithi164.stdout:min_mon_release 0 (unknown) 2020-11-09T12:37:02.134 INFO:teuthology.orchestra.run.smithi164.stdout:election_strategy: 1 2020-11-09T12:37:02.134 INFO:teuthology.orchestra.run.smithi164.stdout:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2020-11-09T12:37:02.135 INFO:teuthology.orchestra.run.smithi164.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-11-09T12:37:02.136 INFO:teuthology.orchestra.run.smithi164:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-09T12:37:02.183 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 77b52c3d-49b9-4299-b194-b448006caee1... 2020-11-09T12:37:02.185 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-11-09T12:37:02.227 INFO:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-11-09T12:37:02.257 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2020-11-09T12:37:02.258 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2020-11-09T12:37:02.258 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-09T12:37:02.258 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2020-11-09T12:37:02.258 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.258 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2020-11-09T12:37:02.258 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.259 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2020-11-09T12:37:02.259 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.259 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2020-11-09T12:37:02.259 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2020-11-09T12:37:02.259 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.259 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2020-11-09T12:37:02.259 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2020-11-09T12:37:02.260 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.260 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-09T12:37:02.260 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2020-11-09T12:37:02.260 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.260 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2020-11-09T12:37:02.260 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-09T12:37:02.260 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.261 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2020-11-09T12:37:02.261 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.261 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2020-11-09T12:37:02.261 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2020-11-09T12:37:02.261 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2020-11-09T12:37:02.261 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2020-11-09T12:37:02.261 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2020-11-09T12:37:02.262 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2020-11-09T12:37:02.262 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2020-11-09T12:37:02.262 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-09T12:37:02.262 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-09T12:37:02.262 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2020-11-09T12:37:02.262 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.262 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-09T12:37:02.263 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.263 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2020-11-09T12:37:02.263 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.263 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2020-11-09T12:37:02.263 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.263 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2020-11-09T12:37:02.263 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2020-11-09T12:37:02.264 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = 172.21.15.164 2020-11-09T12:37:02.264 INFO:teuthology.orchestra.run.smithi164.stdout: osd heartbeat grace = 40 2020-11-09T12:37:02.264 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 77b52c3d-49b9-4299-b194-b448006caee1 2020-11-09T12:37:02.264 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.264 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2020-11-09T12:37:02.264 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2020-11-09T12:37:02.264 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.265 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2020-11-09T12:37:02.265 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2020-11-09T12:37:02.265 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.265 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2020-11-09T12:37:02.265 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2020-11-09T12:37:02.265 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.265 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2020-11-09T12:37:02.266 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2020-11-09T12:37:02.266 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2020-11-09T12:37:02.266 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.266 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2020-11-09T12:37:02.266 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2020-11-09T12:37:02.266 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.266 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2020-11-09T12:37:02.267 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.267 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2020-11-09T12:37:02.267 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.267 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2020-11-09T12:37:02.267 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.267 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2020-11-09T12:37:02.267 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2020-11-09T12:37:02.268 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.268 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2020-11-09T12:37:02.268 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2020-11-09T12:37:02.268 INFO:teuthology.orchestra.run.smithi164.stdout: debug filestore = 20 2020-11-09T12:37:02.268 INFO:teuthology.orchestra.run.smithi164.stdout: debug journal = 20 2020-11-09T12:37:02.270 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 20 2020-11-09T12:37:02.271 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 25 2020-11-09T12:37:02.271 INFO:teuthology.orchestra.run.smithi164.stdout: osd fast shutdown = False 2020-11-09T12:37:02.271 INFO:teuthology.orchestra.run.smithi164.stdout: rocksdb delete range threshold = 0 2020-11-09T12:37:02.271 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.272 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2020-11-09T12:37:02.272 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2020-11-09T12:37:02.272 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2020-11-09T12:37:02.272 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2020-11-09T12:37:02.273 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2020-11-09T12:37:02.273 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2020-11-09T12:37:02.273 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2020-11-09T12:37:02.274 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2020-11-09T12:37:02.274 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.274 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2020-11-09T12:37:02.275 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2020-11-09T12:37:02.275 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2020-11-09T12:37:02.275 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2020-11-09T12:37:02.276 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2020-11-09T12:37:02.276 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2020-11-09T12:37:02.276 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2020-11-09T12:37:02.276 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2020-11-09T12:37:02.277 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2020-11-09T12:37:02.277 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2020-11-09T12:37:02.277 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2020-11-09T12:37:02.278 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd crush smoke test = False 2020-11-09T12:37:02.278 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:02.278 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2020-11-09T12:37:02.278 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2020-11-09T12:37:02.279 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2020-11-09T12:37:02.279 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2020-11-09T12:37:02.279 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-09T12:37:02.279 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-09T12:37:02.280 INFO:teuthology.orchestra.run.smithi164.stdout: debug rgw = 20 2020-11-09T12:37:02.280 INFO:teuthology.orchestra.run.smithi164.stdout: rgw crypt require ssl = False 2020-11-09T12:37:02.280 INFO:teuthology.orchestra.run.smithi164.stdout: rgw crypt s3 kms backend = testing 2020-11-09T12:37:02.280 INFO:teuthology.orchestra.run.smithi164.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-09T12:37:02.281 INFO:teuthology.orchestra.run.smithi164.stdout: rgw curl low speed time = 300 2020-11-09T12:37:02.281 INFO:teuthology.orchestra.run.smithi164.stdout: rgw data log num shards = 4 2020-11-09T12:37:02.281 INFO:teuthology.orchestra.run.smithi164.stdout: rgw md log max shards = 4 2020-11-09T12:37:02.282 INFO:teuthology.orchestra.run.smithi164.stdout: rgw sync log trim interval = 0 2020-11-09T12:37:02.282 INFO:teuthology.orchestra.run.smithi164.stdout: rgw sync obj etag verify = True 2020-11-09T12:37:02.282 INFO:teuthology.orchestra.run.smithi164.stdout: setgroup = ceph 2020-11-09T12:37:02.283 INFO:teuthology.orchestra.run.smithi164.stdout: setuser = ceph 2020-11-09T12:37:02.283 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2020-11-09T12:37:02.284 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.0] 2020-11-09T12:37:02.284 INFO:teuthology.orchestra.run.smithi164.stdout: osd_max_omap_entries_per_request = 10 2020-11-09T12:37:02.285 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.1] 2020-11-09T12:37:02.285 INFO:teuthology.orchestra.run.smithi164.stdout: osd_max_omap_entries_per_request = 1000 2020-11-09T12:37:02.285 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.2] 2020-11-09T12:37:02.285 INFO:teuthology.orchestra.run.smithi164.stdout: osd_max_omap_entries_per_request = 10000 2020-11-09T12:37:02.287 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2020-11-09T12:37:02.288 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2020-11-09T12:37:02.288 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-09T12:37:02.288 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2020-11-09T12:37:02.289 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.289 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2020-11-09T12:37:02.289 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.289 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2020-11-09T12:37:02.290 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.290 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2020-11-09T12:37:02.290 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2020-11-09T12:37:02.290 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.291 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2020-11-09T12:37:02.291 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2020-11-09T12:37:02.291 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.292 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-09T12:37:02.292 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2020-11-09T12:37:02.292 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.292 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2020-11-09T12:37:02.293 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-09T12:37:02.293 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.293 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2020-11-09T12:37:02.293 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.294 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2020-11-09T12:37:02.294 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2020-11-09T12:37:02.294 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2020-11-09T12:37:02.295 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2020-11-09T12:37:02.295 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2020-11-09T12:37:02.295 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2020-11-09T12:37:02.295 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2020-11-09T12:37:02.296 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-09T12:37:02.296 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-09T12:37:02.296 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2020-11-09T12:37:02.296 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.297 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-09T12:37:02.297 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.297 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2020-11-09T12:37:02.297 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.298 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2020-11-09T12:37:02.298 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.298 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2020-11-09T12:37:02.299 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2020-11-09T12:37:02.299 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = 172.21.15.164 2020-11-09T12:37:02.299 INFO:teuthology.orchestra.run.smithi190.stdout: osd heartbeat grace = 40 2020-11-09T12:37:02.299 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = 77b52c3d-49b9-4299-b194-b448006caee1 2020-11-09T12:37:02.300 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.300 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2020-11-09T12:37:02.300 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2020-11-09T12:37:02.300 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.301 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2020-11-09T12:37:02.301 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 600 2020-11-09T12:37:02.301 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.302 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2020-11-09T12:37:02.302 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2020-11-09T12:37:02.302 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.302 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2020-11-09T12:37:02.303 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2020-11-09T12:37:02.303 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2020-11-09T12:37:02.303 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.303 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2020-11-09T12:37:02.304 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2020-11-09T12:37:02.304 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.304 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2020-11-09T12:37:02.304 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.305 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2020-11-09T12:37:02.305 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.305 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2020-11-09T12:37:02.306 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.306 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2020-11-09T12:37:02.306 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2020-11-09T12:37:02.306 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.307 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2020-11-09T12:37:02.307 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2020-11-09T12:37:02.307 INFO:teuthology.orchestra.run.smithi190.stdout: debug filestore = 20 2020-11-09T12:37:02.307 INFO:teuthology.orchestra.run.smithi190.stdout: debug journal = 20 2020-11-09T12:37:02.308 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 20 2020-11-09T12:37:02.308 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 25 2020-11-09T12:37:02.308 INFO:teuthology.orchestra.run.smithi190.stdout: osd fast shutdown = False 2020-11-09T12:37:02.309 INFO:teuthology.orchestra.run.smithi190.stdout: rocksdb delete range threshold = 0 2020-11-09T12:37:02.309 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.309 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2020-11-09T12:37:02.309 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2020-11-09T12:37:02.309 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2020-11-09T12:37:02.310 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2020-11-09T12:37:02.310 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2020-11-09T12:37:02.310 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2020-11-09T12:37:02.310 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2020-11-09T12:37:02.310 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2020-11-09T12:37:02.310 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.311 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2020-11-09T12:37:02.311 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2020-11-09T12:37:02.311 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2020-11-09T12:37:02.311 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2020-11-09T12:37:02.312 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2020-11-09T12:37:02.312 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2020-11-09T12:37:02.312 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2020-11-09T12:37:02.312 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2020-11-09T12:37:02.312 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2020-11-09T12:37:02.312 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2020-11-09T12:37:02.313 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2020-11-09T12:37:02.313 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd crush smoke test = False 2020-11-09T12:37:02.313 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:02.313 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2020-11-09T12:37:02.313 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2020-11-09T12:37:02.313 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2020-11-09T12:37:02.313 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2020-11-09T12:37:02.314 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-09T12:37:02.314 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-09T12:37:02.314 INFO:teuthology.orchestra.run.smithi190.stdout: debug rgw = 20 2020-11-09T12:37:02.314 INFO:teuthology.orchestra.run.smithi190.stdout: rgw crypt require ssl = False 2020-11-09T12:37:02.314 INFO:teuthology.orchestra.run.smithi190.stdout: rgw crypt s3 kms backend = testing 2020-11-09T12:37:02.314 INFO:teuthology.orchestra.run.smithi190.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-09T12:37:02.314 INFO:teuthology.orchestra.run.smithi190.stdout: rgw curl low speed time = 300 2020-11-09T12:37:02.315 INFO:teuthology.orchestra.run.smithi190.stdout: rgw data log num shards = 4 2020-11-09T12:37:02.315 INFO:teuthology.orchestra.run.smithi190.stdout: rgw md log max shards = 4 2020-11-09T12:37:02.315 INFO:teuthology.orchestra.run.smithi190.stdout: rgw sync log trim interval = 0 2020-11-09T12:37:02.315 INFO:teuthology.orchestra.run.smithi190.stdout: rgw sync obj etag verify = True 2020-11-09T12:37:02.315 INFO:teuthology.orchestra.run.smithi190.stdout: setgroup = ceph 2020-11-09T12:37:02.315 INFO:teuthology.orchestra.run.smithi190.stdout: setuser = ceph 2020-11-09T12:37:02.315 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2020-11-09T12:37:02.316 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.0] 2020-11-09T12:37:02.316 INFO:teuthology.orchestra.run.smithi190.stdout: osd_max_omap_entries_per_request = 10 2020-11-09T12:37:02.316 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.1] 2020-11-09T12:37:02.316 INFO:teuthology.orchestra.run.smithi190.stdout: osd_max_omap_entries_per_request = 1000 2020-11-09T12:37:02.316 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.2] 2020-11-09T12:37:02.316 INFO:teuthology.orchestra.run.smithi190.stdout: osd_max_omap_entries_per_request = 10000 2020-11-09T12:37:02.317 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-11-09T12:37:02.317 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-11-09T12:37:02.374 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-09T12:37:02.374 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:02.375 INFO:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-11-09T12:37:02.378 INFO:teuthology.orchestra.run.smithi164.stderr:+ dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-11-09T12:37:02.379 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.] 2020-11-09T12:37:02.379 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDtN6lfSwvtOBAAxG4ArMWzm1HQJL1coCh41g== 2020-11-09T12:37:02.380 INFO:teuthology.orchestra.run.smithi164.stdout:[client.admin] 2020-11-09T12:37:02.380 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDuN6lftNtOFhAAWSdm54QH1fmaaNlFnv0t5A== 2020-11-09T12:37:02.380 INFO:teuthology.orchestra.run.smithi164.stdout: caps mds = "allow *" 2020-11-09T12:37:02.380 INFO:teuthology.orchestra.run.smithi164.stdout: caps mgr = "allow *" 2020-11-09T12:37:02.380 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "allow *" 2020-11-09T12:37:02.380 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "allow *" 2020-11-09T12:37:02.381 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:02.381 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:02.381 INFO:teuthology.orchestra.run.smithi164.stderr:206 bytes copied, 7.3769e-05 s, 2.8 MB/s 2020-11-09T12:37:02.381 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:02.382 INFO:teuthology.orchestra.run.smithi164:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-11-09T12:37:02.425 INFO:teuthology.orchestra.run.smithi164.stderr:+ dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-11-09T12:37:02.426 INFO:teuthology.orchestra.run.smithi164.stdout: �w�,=I�B����Hl���7�_�;�7�_�;�aXa �����a 2020-11-09T12:37:02.427 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:02.427 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:02.427 INFO:teuthology.orchestra.run.smithi164.stderr:201 bytes copied, 0.000164773 s, 1.2 MB/s 2020-11-09T12:37:02.427 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2020-11-09T12:37:02.428 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:02.428 INFO:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/c1.keyring 2020-11-09T12:37:02.428 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-09T12:37:02.473 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-11-09T12:37:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:206 bytes copied, 9.0014e-05 s, 2.3 MB/s 2020-11-09T12:37:02.483 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-09T12:37:02.492 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:02.493 INFO:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-09T12:37:02.537 INFO:teuthology.orchestra.run.smithi164.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-09T12:37:02.538 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:02.539 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:02.539 INFO:teuthology.orchestra.run.smithi164.stderr:201 bytes copied, 0.000121884 s, 1.6 MB/s 2020-11-09T12:37:02.539 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2020-11-09T12:37:02.540 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:02.540 INFO:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/c1.keyring 2020-11-09T12:37:02.540 INFO:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-09T12:37:02.545 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-11-09T12:37:02.554 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:02.555 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:02.555 INFO:teuthology.orchestra.run.smithi190.stderr:206 bytes copied, 6.1094e-05 s, 3.4 MB/s 2020-11-09T12:37:02.556 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-09T12:37:02.565 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:02.565 INFO:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-09T12:37:02.613 INFO:teuthology.orchestra.run.smithi190.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-09T12:37:02.615 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:02.615 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:02.615 INFO:teuthology.orchestra.run.smithi190.stderr:201 bytes copied, 6.5853e-05 s, 3.1 MB/s 2020-11-09T12:37:02.615 INFO:tasks.ceph:Setting up mon nodes... 2020-11-09T12:37:02.616 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-09T12:37:02.616 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-11-09T12:37:02.655 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-11-09T12:37:02.657 INFO:tasks.ceph:Setting up mds nodes... 2020-11-09T12:37:02.658 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-09T12:37:02.658 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-11-09T12:37:02.734 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/c1.client.0.keyring 2020-11-09T12:37:02.745 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-11-09T12:37:02.821 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/c1.client.1.keyring 2020-11-09T12:37:02.833 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-09T12:37:02.834 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-09T12:37:02.834 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-11-09T12:37:02.891 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T12:37:02.892 INFO:tasks.ceph:role: c1.osd.0 2020-11-09T12:37:02.892 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com 2020-11-09T12:37:02.892 INFO:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-09T12:37:03.402 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-09T12:37:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-09T12:37:03.402 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-09T12:37:03.403 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-09T12:37:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2020-11-09T12:37:03.403 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-09T12:37:03.403 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-09T12:37:03.403 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-09T12:37:03.403 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-09T12:37:03.404 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2020-11-09T12:37:03.404 INFO:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2020-11-09T12:37:03.477 INFO:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-11-09T12:37:03.488 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: /sbin/restorecon: command not found 2020-11-09T12:37:03.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-09T12:37:03.488 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-11-09T12:37:03.543 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T12:37:03.544 INFO:tasks.ceph:role: c1.osd.1 2020-11-09T12:37:03.544 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com 2020-11-09T12:37:03.544 INFO:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-09T12:37:04.061 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-09T12:37:04.061 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-09T12:37:04.062 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-09T12:37:04.062 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-09T12:37:04.062 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2020-11-09T12:37:04.062 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-09T12:37:04.062 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-09T12:37:04.062 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-09T12:37:04.063 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-09T12:37:04.063 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2020-11-09T12:37:04.064 INFO:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2020-11-09T12:37:04.138 INFO:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-11-09T12:37:04.155 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: /sbin/restorecon: command not found 2020-11-09T12:37:04.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-09T12:37:04.156 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-11-09T12:37:04.212 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T12:37:04.212 INFO:tasks.ceph:role: c1.osd.2 2020-11-09T12:37:04.213 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com 2020-11-09T12:37:04.213 INFO:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-09T12:37:04.716 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-09T12:37:04.716 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-09T12:37:04.716 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-09T12:37:04.716 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-09T12:37:04.717 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2020-11-09T12:37:04.717 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-09T12:37:04.717 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-09T12:37:04.717 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-09T12:37:04.717 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-09T12:37:04.718 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2020-11-09T12:37:04.718 INFO:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2020-11-09T12:37:04.776 INFO:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-11-09T12:37:04.787 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: /sbin/restorecon: command not found 2020-11-09T12:37:04.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-09T12:37:04.788 INFO:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-09T12:37:04.874 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:04.875+0000 7f0bb83a5d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-11-09T12:37:04.874 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:04.875+0000 7f0bb83a5d80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-11-09T12:37:04.874 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:04.875+0000 7f0bb83a5d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-09T12:37:04.874 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:04.875+0000 7f0bb83a5d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-09T12:37:04.875 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:04.875+0000 7f0bb83a5d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-09T12:37:04.882 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:04.883+0000 7f0bb83a5d80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-11-09T12:37:07.045 INFO:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-11-09T12:37:07.061 INFO:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-09T12:37:07.144 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:07.143+0000 7f8528bcbd80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-11-09T12:37:07.144 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:07.143+0000 7f8528bcbd80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-11-09T12:37:07.145 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:07.143+0000 7f8528bcbd80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-09T12:37:07.145 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:07.143+0000 7f8528bcbd80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-09T12:37:07.145 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:07.143+0000 7f8528bcbd80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-09T12:37:07.146 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:07.143+0000 7f8528bcbd80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-11-09T12:37:09.324 INFO:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-11-09T12:37:09.380 INFO:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-09T12:37:09.462 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:09.463+0000 7fab7fcdad80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-11-09T12:37:09.462 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:09.463+0000 7fab7fcdad80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-11-09T12:37:09.462 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:09.463+0000 7fab7fcdad80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-09T12:37:09.462 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:09.463+0000 7fab7fcdad80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-09T12:37:09.463 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:09.463+0000 7fab7fcdad80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-09T12:37:09.463 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:37:09.463+0000 7fab7fcdad80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-11-09T12:37:11.636 INFO:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-11-09T12:37:11.692 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-09T12:37:11.693 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:11.693 INFO:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-11-09T12:37:11.737 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-11-09T12:37:11.746 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.x] 2020-11-09T12:37:11.746 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDuN6lfO3E6JxAAnGawem0Xx9CXITd/S7GyQQ== 2020-11-09T12:37:11.746 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:11.746 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:11.746 INFO:teuthology.orchestra.run.smithi164.stderr:56 bytes copied, 0.000159077 s, 352 kB/s 2020-11-09T12:37:11.747 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:11.748 INFO:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-11-09T12:37:11.793 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-11-09T12:37:11.800 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.0] 2020-11-09T12:37:11.800 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDwN6lfTrI6NBAAxpg6B9D8T61hn/lwv7jrWA== 2020-11-09T12:37:11.800 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:11.801 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:11.801 INFO:teuthology.orchestra.run.smithi164.stderr:56 bytes copied, 0.000159703 s, 351 kB/s 2020-11-09T12:37:11.803 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:11.803 INFO:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-11-09T12:37:11.849 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-11-09T12:37:11.857 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.1] 2020-11-09T12:37:11.858 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDzN6lf2ZS0CBAA5e30NJGs57hKV3DNRzF9yg== 2020-11-09T12:37:11.858 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:11.858 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:11.858 INFO:teuthology.orchestra.run.smithi164.stderr:56 bytes copied, 0.000156727 s, 357 kB/s 2020-11-09T12:37:11.859 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:11.860 INFO:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-11-09T12:37:11.905 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-11-09T12:37:11.914 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.2] 2020-11-09T12:37:11.914 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQD1N6lfkcesGxAAZXM9meDwPJDO/mk6DuVvPA== 2020-11-09T12:37:11.914 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:11.914 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:11.915 INFO:teuthology.orchestra.run.smithi164.stderr:56 bytes copied, 0.0001587 s, 353 kB/s 2020-11-09T12:37:11.915 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:11.916 INFO:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-11-09T12:37:11.961 INFO:teuthology.orchestra.run.smithi164.stderr:+ dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-11-09T12:37:11.962 INFO:teuthology.orchestra.run.smithi164.stdout:[client.0] 2020-11-09T12:37:11.962 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDuN6lfNannKxAAoTxRnNz8kcHGDkfukg2F1Q== 2020-11-09T12:37:11.963 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:11.963 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:11.963 INFO:teuthology.orchestra.run.smithi164.stderr:59 bytes copied, 0.00016006 s, 369 kB/s 2020-11-09T12:37:11.963 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:11.963 INFO:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-11-09T12:37:12.009 INFO:teuthology.orchestra.run.smithi164.stderr:+ dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-11-09T12:37:12.010 INFO:teuthology.orchestra.run.smithi164.stdout:[client.1] 2020-11-09T12:37:12.010 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDuN6lf6wAkMRAAe+5l8F34B3/gpa98/29fJA== 2020-11-09T12:37:12.011 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:12.011 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:12.011 INFO:teuthology.orchestra.run.smithi164.stderr:59 bytes copied, 0.00016173 s, 365 kB/s 2020-11-09T12:37:12.011 INFO:tasks.ceph:Adding keys to all mons... 2020-11-09T12:37:12.011 INFO:teuthology.orchestra.run.smithi164:> sudo tee -a /etc/ceph/c1.keyring 2020-11-09T12:37:12.066 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.x] 2020-11-09T12:37:12.066 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDuN6lfO3E6JxAAnGawem0Xx9CXITd/S7GyQQ== 2020-11-09T12:37:12.066 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.0] 2020-11-09T12:37:12.066 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDwN6lfTrI6NBAAxpg6B9D8T61hn/lwv7jrWA== 2020-11-09T12:37:12.066 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.1] 2020-11-09T12:37:12.067 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDzN6lf2ZS0CBAA5e30NJGs57hKV3DNRzF9yg== 2020-11-09T12:37:12.067 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.2] 2020-11-09T12:37:12.067 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQD1N6lfkcesGxAAZXM9meDwPJDO/mk6DuVvPA== 2020-11-09T12:37:12.067 INFO:teuthology.orchestra.run.smithi164.stdout:[client.0] 2020-11-09T12:37:12.067 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDuN6lfNannKxAAoTxRnNz8kcHGDkfukg2F1Q== 2020-11-09T12:37:12.067 INFO:teuthology.orchestra.run.smithi164.stdout:[client.1] 2020-11-09T12:37:12.067 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDuN6lf6wAkMRAAe+5l8F34B3/gpa98/29fJA== 2020-11-09T12:37:12.068 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-09T12:37:12.153 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-09T12:37:12.233 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-09T12:37:12.318 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-09T12:37:12.405 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-09T12:37:12.495 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-09T12:37:12.585 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-09T12:37:12.585 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-11-09T12:37:12.644 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-11-09T12:37:12.739 INFO:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-11-09T12:37:12.752 INFO:teuthology.orchestra.run.smithi164:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-11-09T12:37:12.798 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-11-09T12:37:12.799 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-11-09T12:37:12.799 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2020-11-09T12:37:12.842 INFO:tasks.ceph.c1.mon.a:Started 2020-11-09T12:37:12.843 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-11-09T12:37:12.843 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-11-09T12:37:12.844 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2020-11-09T12:37:12.847 INFO:tasks.ceph.c1.mgr.x:Started 2020-11-09T12:37:12.848 INFO:tasks.ceph:Setting crush tunables to default 2020-11-09T12:37:12.849 INFO:teuthology.orchestra.run.smithi164:> sudo ceph --cluster c1 osd crush tunables default 2020-11-09T12:37:13.194 INFO:teuthology.orchestra.run.smithi164.stderr:adjusted tunables profile to default 2020-11-09T12:37:13.206 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-11-09T12:37:13.207 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:13.207 INFO:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-11-09T12:37:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-11-09T12:37:13.218 INFO:teuthology.orchestra.run.smithi164.stdout:b8a39758-639b-403e-b6ca-63b45e00f430 2020-11-09T12:37:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:37 bytes copied, 6.8077e-05 s, 544 kB/s 2020-11-09T12:37:13.219 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:13.220 INFO:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-11-09T12:37:13.265 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-11-09T12:37:13.272 INFO:teuthology.orchestra.run.smithi164.stdout:9df8d09d-e31b-4f29-9944-2e765de4b005 2020-11-09T12:37:13.272 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:13.272 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:13.272 INFO:teuthology.orchestra.run.smithi164.stderr:37 bytes copied, 0.000133432 s, 277 kB/s 2020-11-09T12:37:13.273 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:13.274 INFO:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-11-09T12:37:13.321 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-11-09T12:37:13.328 INFO:teuthology.orchestra.run.smithi164.stdout:6961ba66-a954-4d09-acd2-abbb10b615e9 2020-11-09T12:37:13.329 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:13.329 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:13.329 INFO:teuthology.orchestra.run.smithi164.stderr:37 bytes copied, 0.000125 s, 296 kB/s 2020-11-09T12:37:13.330 INFO:teuthology.orchestra.run.smithi164:> sudo ceph --cluster c1 osd new b8a39758-639b-403e-b6ca-63b45e00f430 0 2020-11-09T12:37:13.591 INFO:teuthology.orchestra.run.smithi164.stdout:0 2020-11-09T12:37:13.605 INFO:teuthology.orchestra.run.smithi164:> sudo ceph --cluster c1 osd new 9df8d09d-e31b-4f29-9944-2e765de4b005 1 2020-11-09T12:37:13.822 INFO:teuthology.orchestra.run.smithi164.stdout:1 2020-11-09T12:37:13.834 INFO:teuthology.orchestra.run.smithi164:> sudo ceph --cluster c1 osd new 6961ba66-a954-4d09-acd2-abbb10b615e9 2 2020-11-09T12:37:14.048 INFO:teuthology.orchestra.run.smithi164.stdout:2 2020-11-09T12:37:14.061 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-11-09T12:37:14.062 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2020-11-09T12:37:14.064 INFO:tasks.ceph.c1.osd.0:Started 2020-11-09T12:37:14.064 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-11-09T12:37:14.065 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2020-11-09T12:37:14.067 INFO:tasks.ceph.c1.osd.1:Started 2020-11-09T12:37:14.067 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-11-09T12:37:14.068 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2020-11-09T12:37:14.070 INFO:tasks.ceph.c1.osd.2:Started 2020-11-09T12:37:14.070 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-09T12:37:14.374 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-11-09T12:37:14.447 INFO:tasks.ceph.c1.osd.1.smithi164.stderr:2020-11-09T12:37:14.447+0000 7f8b90809d80 -1 Falling back to public interface 2020-11-09T12:37:14.549 INFO:teuthology.misc.health.smithi164.stdout: 2020-11-09T12:37:14.550 INFO:teuthology.misc.health.smithi164.stdout:{"epoch":5,"fsid":"77b52c3d-49b9-4299-b194-b448006caee1","created":"2020-11-09T12:37:12.961910+0000","modified":"2020-11-09T12:37:14.048974+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"b8a39758-639b-403e-b6ca-63b45e00f430","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9df8d09d-e31b-4f29-9944-2e765de4b005","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6961ba66-a954-4d09-acd2-abbb10b615e9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T12:37:14.561 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-11-09T12:37:14.671 INFO:tasks.ceph.c1.osd.0.smithi164.stderr:2020-11-09T12:37:14.671+0000 7fd8afb86d80 -1 Falling back to public interface 2020-11-09T12:37:14.683 INFO:tasks.ceph.c1.osd.2.smithi164.stderr:2020-11-09T12:37:14.683+0000 7f658c34ed80 -1 Falling back to public interface 2020-11-09T12:37:15.096 INFO:tasks.ceph.c1.osd.1.smithi164.stderr:2020-11-09T12:37:15.095+0000 7f8b90809d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-09T12:37:15.285 INFO:tasks.ceph.c1.osd.0.smithi164.stderr:2020-11-09T12:37:15.287+0000 7fd8afb86d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-09T12:37:15.295 INFO:tasks.ceph.c1.osd.2.smithi164.stderr:2020-11-09T12:37:15.295+0000 7f658c34ed80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-09T12:37:16.988 INFO:tasks.ceph.c1.osd.1.smithi164.stderr:2020-11-09T12:37:16.987+0000 7f8b87cd3700 -1 osd.1 0 waiting for initial osdmap 2020-11-09T12:37:16.988 INFO:tasks.ceph.c1.osd.0.smithi164.stderr:2020-11-09T12:37:16.987+0000 7fd8a7050700 -1 osd.0 0 waiting for initial osdmap 2020-11-09T12:37:16.989 INFO:tasks.ceph.c1.osd.2.smithi164.stderr:2020-11-09T12:37:16.991+0000 7f6583818700 -1 osd.2 0 waiting for initial osdmap 2020-11-09T12:37:20.865 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-11-09T12:37:21.147 INFO:teuthology.misc.health.smithi164.stdout: 2020-11-09T12:37:21.148 INFO:teuthology.misc.health.smithi164.stdout:{"epoch":10,"fsid":"77b52c3d-49b9-4299-b194-b448006caee1","created":"2020-11-09T12:37:12.961910+0000","modified":"2020-11-09T12:37:20.007315+0000","last_up_change":"2020-11-09T12:37:17.984398+0000","last_in_change":"2020-11-09T12:37:17.984398+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T12:37:16.908400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"b8a39758-639b-403e-b6ca-63b45e00f430","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":12348},{"type":"v1","addr":"172.21.15.164:6809","nonce":12348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":12348},{"type":"v1","addr":"172.21.15.164:6811","nonce":12348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":12348},{"type":"v1","addr":"172.21.15.164:6815","nonce":12348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":12348},{"type":"v1","addr":"172.21.15.164:6813","nonce":12348}]},"public_addr":"172.21.15.164:6809/12348","cluster_addr":"172.21.15.164:6811/12348","heartbeat_back_addr":"172.21.15.164:6815/12348","heartbeat_front_addr":"172.21.15.164:6813/12348","state":["exists","up"]},{"osd":1,"uuid":"9df8d09d-e31b-4f29-9944-2e765de4b005","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":12347},{"type":"v1","addr":"172.21.15.164:6801","nonce":12347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":12347},{"type":"v1","addr":"172.21.15.164:6803","nonce":12347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":12347},{"type":"v1","addr":"172.21.15.164:6807","nonce":12347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":12347},{"type":"v1","addr":"172.21.15.164:6805","nonce":12347}]},"public_addr":"172.21.15.164:6801/12347","cluster_addr":"172.21.15.164:6803/12347","heartbeat_back_addr":"172.21.15.164:6807/12347","heartbeat_front_addr":"172.21.15.164:6805/12347","state":["exists","up"]},{"osd":2,"uuid":"6961ba66-a954-4d09-acd2-abbb10b615e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":12349},{"type":"v1","addr":"172.21.15.164:6817","nonce":12349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":12349},{"type":"v1","addr":"172.21.15.164:6819","nonce":12349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":12349},{"type":"v1","addr":"172.21.15.164:6823","nonce":12349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":12349},{"type":"v1","addr":"172.21.15.164:6821","nonce":12349}]},"public_addr":"172.21.15.164:6817/12349","cluster_addr":"172.21.15.164:6819/12349","heartbeat_back_addr":"172.21.15.164:6823/12349","heartbeat_front_addr":"172.21.15.164:6821/12349","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:16.259812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:16.122020+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:16.304224+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T12:37:21.160 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-11-09T12:37:21.161 INFO:tasks.ceph:Creating RBD pool 2020-11-09T12:37:21.161 INFO:teuthology.orchestra.run.smithi164:> sudo ceph --cluster c1 osd pool create rbd 8 2020-11-09T12:37:22.043 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'rbd' created 2020-11-09T12:37:22.057 INFO:teuthology.orchestra.run.smithi164:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-09T12:37:23.051 INFO:teuthology.orchestra.run.smithi164.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-09T12:37:23.064 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-11-09T12:37:23.065 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-09T12:37:23.066 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-09T12:37:23.354 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:23.355 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":12,"fsid":"77b52c3d-49b9-4299-b194-b448006caee1","created":"2020-11-09T12:37:12.961910+0000","modified":"2020-11-09T12:37:23.030578+0000","last_up_change":"2020-11-09T12:37:17.984398+0000","last_in_change":"2020-11-09T12:37:17.984398+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T12:37:16.908400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-09T12:37:21.447685+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b8a39758-639b-403e-b6ca-63b45e00f430","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":12348},{"type":"v1","addr":"172.21.15.164:6809","nonce":12348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":12348},{"type":"v1","addr":"172.21.15.164:6811","nonce":12348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":12348},{"type":"v1","addr":"172.21.15.164:6815","nonce":12348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":12348},{"type":"v1","addr":"172.21.15.164:6813","nonce":12348}]},"public_addr":"172.21.15.164:6809/12348","cluster_addr":"172.21.15.164:6811/12348","heartbeat_back_addr":"172.21.15.164:6815/12348","heartbeat_front_addr":"172.21.15.164:6813/12348","state":["exists","up"]},{"osd":1,"uuid":"9df8d09d-e31b-4f29-9944-2e765de4b005","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":12347},{"type":"v1","addr":"172.21.15.164:6801","nonce":12347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":12347},{"type":"v1","addr":"172.21.15.164:6803","nonce":12347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":12347},{"type":"v1","addr":"172.21.15.164:6807","nonce":12347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":12347},{"type":"v1","addr":"172.21.15.164:6805","nonce":12347}]},"public_addr":"172.21.15.164:6801/12347","cluster_addr":"172.21.15.164:6803/12347","heartbeat_back_addr":"172.21.15.164:6807/12347","heartbeat_front_addr":"172.21.15.164:6805/12347","state":["exists","up"]},{"osd":2,"uuid":"6961ba66-a954-4d09-acd2-abbb10b615e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":12349},{"type":"v1","addr":"172.21.15.164:6817","nonce":12349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":12349},{"type":"v1","addr":"172.21.15.164:6819","nonce":12349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":12349},{"type":"v1","addr":"172.21.15.164:6823","nonce":12349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":12349},{"type":"v1","addr":"172.21.15.164:6821","nonce":12349}]},"public_addr":"172.21.15.164:6817/12349","cluster_addr":"172.21.15.164:6819/12349","heartbeat_back_addr":"172.21.15.164:6823/12349","heartbeat_front_addr":"172.21.15.164:6821/12349","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:16.259812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:16.122020+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:16.304224+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T12:37:23.369 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-09T12:37:16.908400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-09T12:37:21.447685+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-09T12:37:23.369 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get device_health_metrics pg_num 2020-11-09T12:37:23.666 INFO:teuthology.orchestra.run.smithi164.stdout:pg_num: 1 2020-11-09T12:37:23.680 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2020-11-09T12:37:23.966 INFO:teuthology.orchestra.run.smithi164.stdout:pg_num: 8 2020-11-09T12:37:23.981 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-11-09T12:37:23.981 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-11-09T12:37:23.982 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2020-11-09T12:37:24.313 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:24.327 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":12166},{"type":"v1","addr":"172.21.15.164:6825","nonce":12166}]},"active_addr":"172.21.15.164:6825/12166","active_change":"2020-11-09T12:37:16.877303+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":2785298013}]},{"addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":3470721935}]},{"addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":1170164191}]}]}} 2020-11-09T12:37:24.332 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-11-09T12:37:24.332 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2020-11-09T12:37:24.333 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-09T12:37:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:24.617 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":13,"fsid":"77b52c3d-49b9-4299-b194-b448006caee1","created":"2020-11-09T12:37:12.961910+0000","modified":"2020-11-09T12:37:24.044206+0000","last_up_change":"2020-11-09T12:37:17.984398+0000","last_in_change":"2020-11-09T12:37:17.984398+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T12:37:16.908400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-09T12:37:21.447685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b8a39758-639b-403e-b6ca-63b45e00f430","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":12348},{"type":"v1","addr":"172.21.15.164:6809","nonce":12348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":12348},{"type":"v1","addr":"172.21.15.164:6811","nonce":12348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":12348},{"type":"v1","addr":"172.21.15.164:6815","nonce":12348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":12348},{"type":"v1","addr":"172.21.15.164:6813","nonce":12348}]},"public_addr":"172.21.15.164:6809/12348","cluster_addr":"172.21.15.164:6811/12348","heartbeat_back_addr":"172.21.15.164:6815/12348","heartbeat_front_addr":"172.21.15.164:6813/12348","state":["exists","up"]},{"osd":1,"uuid":"9df8d09d-e31b-4f29-9944-2e765de4b005","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":12347},{"type":"v1","addr":"172.21.15.164:6801","nonce":12347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":12347},{"type":"v1","addr":"172.21.15.164:6803","nonce":12347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":12347},{"type":"v1","addr":"172.21.15.164:6807","nonce":12347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":12347},{"type":"v1","addr":"172.21.15.164:6805","nonce":12347}]},"public_addr":"172.21.15.164:6801/12347","cluster_addr":"172.21.15.164:6803/12347","heartbeat_back_addr":"172.21.15.164:6807/12347","heartbeat_front_addr":"172.21.15.164:6805/12347","state":["exists","up"]},{"osd":2,"uuid":"6961ba66-a954-4d09-acd2-abbb10b615e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":12349},{"type":"v1","addr":"172.21.15.164:6817","nonce":12349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":12349},{"type":"v1","addr":"172.21.15.164:6819","nonce":12349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":12349},{"type":"v1","addr":"172.21.15.164:6823","nonce":12349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":12349},{"type":"v1","addr":"172.21.15.164:6821","nonce":12349}]},"public_addr":"172.21.15.164:6817/12349","cluster_addr":"172.21.15.164:6819/12349","heartbeat_back_addr":"172.21.15.164:6823/12349","heartbeat_front_addr":"172.21.15.164:6821/12349","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:16.259812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:16.122020+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:16.304224+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T12:37:24.631 INFO:tasks.ceph.ceph_manager.c1:all up! 2020-11-09T12:37:24.631 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-09T12:37:24.913 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:24.913 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":13,"fsid":"77b52c3d-49b9-4299-b194-b448006caee1","created":"2020-11-09T12:37:12.961910+0000","modified":"2020-11-09T12:37:24.044206+0000","last_up_change":"2020-11-09T12:37:17.984398+0000","last_in_change":"2020-11-09T12:37:17.984398+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T12:37:16.908400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-09T12:37:21.447685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b8a39758-639b-403e-b6ca-63b45e00f430","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":12348},{"type":"v1","addr":"172.21.15.164:6809","nonce":12348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":12348},{"type":"v1","addr":"172.21.15.164:6811","nonce":12348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":12348},{"type":"v1","addr":"172.21.15.164:6815","nonce":12348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":12348},{"type":"v1","addr":"172.21.15.164:6813","nonce":12348}]},"public_addr":"172.21.15.164:6809/12348","cluster_addr":"172.21.15.164:6811/12348","heartbeat_back_addr":"172.21.15.164:6815/12348","heartbeat_front_addr":"172.21.15.164:6813/12348","state":["exists","up"]},{"osd":1,"uuid":"9df8d09d-e31b-4f29-9944-2e765de4b005","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":12347},{"type":"v1","addr":"172.21.15.164:6801","nonce":12347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":12347},{"type":"v1","addr":"172.21.15.164:6803","nonce":12347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":12347},{"type":"v1","addr":"172.21.15.164:6807","nonce":12347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":12347},{"type":"v1","addr":"172.21.15.164:6805","nonce":12347}]},"public_addr":"172.21.15.164:6801/12347","cluster_addr":"172.21.15.164:6803/12347","heartbeat_back_addr":"172.21.15.164:6807/12347","heartbeat_front_addr":"172.21.15.164:6805/12347","state":["exists","up"]},{"osd":2,"uuid":"6961ba66-a954-4d09-acd2-abbb10b615e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":12349},{"type":"v1","addr":"172.21.15.164:6817","nonce":12349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":12349},{"type":"v1","addr":"172.21.15.164:6819","nonce":12349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":12349},{"type":"v1","addr":"172.21.15.164:6823","nonce":12349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":12349},{"type":"v1","addr":"172.21.15.164:6821","nonce":12349}]},"public_addr":"172.21.15.164:6817/12349","cluster_addr":"172.21.15.164:6819/12349","heartbeat_back_addr":"172.21.15.164:6823/12349","heartbeat_front_addr":"172.21.15.164:6821/12349","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:16.259812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:16.122020+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:16.304224+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T12:37:24.927 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2020-11-09T12:37:25.098 INFO:teuthology.orchestra.run.smithi164.stdout:34359738371 2020-11-09T12:37:25.098 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2020-11-09T12:37:25.260 INFO:teuthology.orchestra.run.smithi164.stdout:34359738371 2020-11-09T12:37:25.260 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2020-11-09T12:37:25.425 INFO:teuthology.orchestra.run.smithi164.stdout:34359738372 2020-11-09T12:37:25.426 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2020-11-09T12:37:25.722 INFO:teuthology.orchestra.run.smithi164.stdout:34359738370 2020-11-09T12:37:25.736 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2020-11-09T12:37:26.737 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2020-11-09T12:37:27.030 INFO:teuthology.orchestra.run.smithi164.stdout:34359738370 2020-11-09T12:37:27.044 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2020-11-09T12:37:28.046 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2020-11-09T12:37:28.348 INFO:teuthology.orchestra.run.smithi164.stdout:34359738371 2020-11-09T12:37:28.362 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2020-11-09T12:37:28.363 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2020-11-09T12:37:28.649 INFO:teuthology.orchestra.run.smithi164.stdout:34359738371 2020-11-09T12:37:28.663 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.1 2020-11-09T12:37:28.664 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2020-11-09T12:37:28.954 INFO:teuthology.orchestra.run.smithi164.stdout:34359738372 2020-11-09T12:37:28.969 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2020-11-09T12:37:28.969 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-11-09T12:37:28.969 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2020-11-09T12:37:29.263 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:29.263 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2020-11-09T12:37:29.269 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-09T12:37:28.888211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14752,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558048,"statfs":{"total":322122547200,"available":322107441152,"internally_reserved":0,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.841643"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.055144+0000","last_change":"2020-11-09T12:37:23.052410+0000","last_active":"2020-11-09T12:37:23.055144+0000","last_peered":"2020-11-09T12:37:23.055144+0000","last_clean":"2020-11-09T12:37:23.055144+0000","last_became_active":"2020-11-09T12:37:23.051194+0000","last_became_peered":"2020-11-09T12:37:23.051194+0000","last_unstale":"2020-11-09T12:37:23.055144+0000","last_undegraded":"2020-11-09T12:37:23.055144+0000","last_fullsized":"2020-11-09T12:37:23.055144+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.055344+0000","last_change":"2020-11-09T12:37:23.052255+0000","last_active":"2020-11-09T12:37:23.055344+0000","last_peered":"2020-11-09T12:37:23.055344+0000","last_clean":"2020-11-09T12:37:23.055344+0000","last_became_active":"2020-11-09T12:37:23.050847+0000","last_became_peered":"2020-11-09T12:37:23.050847+0000","last_unstale":"2020-11-09T12:37:23.055344+0000","last_undegraded":"2020-11-09T12:37:23.055344+0000","last_fullsized":"2020-11-09T12:37:23.055344+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.055407+0000","last_change":"2020-11-09T12:37:23.053987+0000","last_active":"2020-11-09T12:37:23.055407+0000","last_peered":"2020-11-09T12:37:23.055407+0000","last_clean":"2020-11-09T12:37:23.055407+0000","last_became_active":"2020-11-09T12:37:23.053560+0000","last_became_peered":"2020-11-09T12:37:23.053560+0000","last_unstale":"2020-11-09T12:37:23.055407+0000","last_undegraded":"2020-11-09T12:37:23.055407+0000","last_fullsized":"2020-11-09T12:37:23.055407+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.051757+0000","last_change":"2020-11-09T12:37:23.048860+0000","last_active":"2020-11-09T12:37:23.051757+0000","last_peered":"2020-11-09T12:37:23.051757+0000","last_clean":"2020-11-09T12:37:23.051757+0000","last_became_active":"2020-11-09T12:37:23.047156+0000","last_became_peered":"2020-11-09T12:37:23.047156+0000","last_unstale":"2020-11-09T12:37:23.051757+0000","last_undegraded":"2020-11-09T12:37:23.051757+0000","last_fullsized":"2020-11-09T12:37:23.051757+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.046693+0000","last_change":"2020-11-09T12:37:19.270828+0000","last_active":"2020-11-09T12:37:23.046693+0000","last_peered":"2020-11-09T12:37:23.046693+0000","last_clean":"2020-11-09T12:37:23.046693+0000","last_became_active":"2020-11-09T12:37:19.269236+0000","last_became_peered":"2020-11-09T12:37:19.269236+0000","last_unstale":"2020-11-09T12:37:23.046693+0000","last_undegraded":"2020-11-09T12:37:23.046693+0000","last_fullsized":"2020-11-09T12:37:23.046693+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:16.976871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:16.976871+0000","last_clean_scrub_stamp":"2020-11-09T12:37:16.976871+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.052066+0000","last_change":"2020-11-09T12:37:23.049760+0000","last_active":"2020-11-09T12:37:23.052066+0000","last_peered":"2020-11-09T12:37:23.052066+0000","last_clean":"2020-11-09T12:37:23.052066+0000","last_became_active":"2020-11-09T12:37:23.049446+0000","last_became_peered":"2020-11-09T12:37:23.049446+0000","last_unstale":"2020-11-09T12:37:23.052066+0000","last_undegraded":"2020-11-09T12:37:23.052066+0000","last_fullsized":"2020-11-09T12:37:23.052066+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.051819+0000","last_change":"2020-11-09T12:37:23.049073+0000","last_active":"2020-11-09T12:37:23.051819+0000","last_peered":"2020-11-09T12:37:23.051819+0000","last_clean":"2020-11-09T12:37:23.051819+0000","last_became_active":"2020-11-09T12:37:23.048572+0000","last_became_peered":"2020-11-09T12:37:23.048572+0000","last_unstale":"2020-11-09T12:37:23.051819+0000","last_undegraded":"2020-11-09T12:37:23.051819+0000","last_fullsized":"2020-11-09T12:37:23.051819+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.057046+0000","last_change":"2020-11-09T12:37:23.055451+0000","last_active":"2020-11-09T12:37:23.057046+0000","last_peered":"2020-11-09T12:37:23.057046+0000","last_clean":"2020-11-09T12:37:23.057046+0000","last_became_active":"2020-11-09T12:37:23.055097+0000","last_became_peered":"2020-11-09T12:37:23.055097+0000","last_unstale":"2020-11-09T12:37:23.057046+0000","last_undegraded":"2020-11-09T12:37:23.057046+0000","last_fullsized":"2020-11-09T12:37:23.057046+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.055270+0000","last_change":"2020-11-09T12:37:23.052973+0000","last_active":"2020-11-09T12:37:23.055270+0000","last_peered":"2020-11-09T12:37:23.055270+0000","last_clean":"2020-11-09T12:37:23.055270+0000","last_became_active":"2020-11-09T12:37:23.051846+0000","last_became_peered":"2020-11-09T12:37:23.051846+0000","last_unstale":"2020-11-09T12:37:23.055270+0000","last_undegraded":"2020-11-09T12:37:23.055270+0000","last_fullsized":"2020-11-09T12:37:23.055270+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.373}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.603}]}]}]}} 2020-11-09T12:37:29.270 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2020-11-09T12:37:29.578 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:29.578 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2020-11-09T12:37:29.590 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-09T12:37:28.888211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14752,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558048,"statfs":{"total":322122547200,"available":322107441152,"internally_reserved":0,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.841643"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.055144+0000","last_change":"2020-11-09T12:37:23.052410+0000","last_active":"2020-11-09T12:37:23.055144+0000","last_peered":"2020-11-09T12:37:23.055144+0000","last_clean":"2020-11-09T12:37:23.055144+0000","last_became_active":"2020-11-09T12:37:23.051194+0000","last_became_peered":"2020-11-09T12:37:23.051194+0000","last_unstale":"2020-11-09T12:37:23.055144+0000","last_undegraded":"2020-11-09T12:37:23.055144+0000","last_fullsized":"2020-11-09T12:37:23.055144+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.055344+0000","last_change":"2020-11-09T12:37:23.052255+0000","last_active":"2020-11-09T12:37:23.055344+0000","last_peered":"2020-11-09T12:37:23.055344+0000","last_clean":"2020-11-09T12:37:23.055344+0000","last_became_active":"2020-11-09T12:37:23.050847+0000","last_became_peered":"2020-11-09T12:37:23.050847+0000","last_unstale":"2020-11-09T12:37:23.055344+0000","last_undegraded":"2020-11-09T12:37:23.055344+0000","last_fullsized":"2020-11-09T12:37:23.055344+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.055407+0000","last_change":"2020-11-09T12:37:23.053987+0000","last_active":"2020-11-09T12:37:23.055407+0000","last_peered":"2020-11-09T12:37:23.055407+0000","last_clean":"2020-11-09T12:37:23.055407+0000","last_became_active":"2020-11-09T12:37:23.053560+0000","last_became_peered":"2020-11-09T12:37:23.053560+0000","last_unstale":"2020-11-09T12:37:23.055407+0000","last_undegraded":"2020-11-09T12:37:23.055407+0000","last_fullsized":"2020-11-09T12:37:23.055407+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.051757+0000","last_change":"2020-11-09T12:37:23.048860+0000","last_active":"2020-11-09T12:37:23.051757+0000","last_peered":"2020-11-09T12:37:23.051757+0000","last_clean":"2020-11-09T12:37:23.051757+0000","last_became_active":"2020-11-09T12:37:23.047156+0000","last_became_peered":"2020-11-09T12:37:23.047156+0000","last_unstale":"2020-11-09T12:37:23.051757+0000","last_undegraded":"2020-11-09T12:37:23.051757+0000","last_fullsized":"2020-11-09T12:37:23.051757+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.046693+0000","last_change":"2020-11-09T12:37:19.270828+0000","last_active":"2020-11-09T12:37:23.046693+0000","last_peered":"2020-11-09T12:37:23.046693+0000","last_clean":"2020-11-09T12:37:23.046693+0000","last_became_active":"2020-11-09T12:37:19.269236+0000","last_became_peered":"2020-11-09T12:37:19.269236+0000","last_unstale":"2020-11-09T12:37:23.046693+0000","last_undegraded":"2020-11-09T12:37:23.046693+0000","last_fullsized":"2020-11-09T12:37:23.046693+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:16.976871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:16.976871+0000","last_clean_scrub_stamp":"2020-11-09T12:37:16.976871+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.052066+0000","last_change":"2020-11-09T12:37:23.049760+0000","last_active":"2020-11-09T12:37:23.052066+0000","last_peered":"2020-11-09T12:37:23.052066+0000","last_clean":"2020-11-09T12:37:23.052066+0000","last_became_active":"2020-11-09T12:37:23.049446+0000","last_became_peered":"2020-11-09T12:37:23.049446+0000","last_unstale":"2020-11-09T12:37:23.052066+0000","last_undegraded":"2020-11-09T12:37:23.052066+0000","last_fullsized":"2020-11-09T12:37:23.052066+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.051819+0000","last_change":"2020-11-09T12:37:23.049073+0000","last_active":"2020-11-09T12:37:23.051819+0000","last_peered":"2020-11-09T12:37:23.051819+0000","last_clean":"2020-11-09T12:37:23.051819+0000","last_became_active":"2020-11-09T12:37:23.048572+0000","last_became_peered":"2020-11-09T12:37:23.048572+0000","last_unstale":"2020-11-09T12:37:23.051819+0000","last_undegraded":"2020-11-09T12:37:23.051819+0000","last_fullsized":"2020-11-09T12:37:23.051819+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.057046+0000","last_change":"2020-11-09T12:37:23.055451+0000","last_active":"2020-11-09T12:37:23.057046+0000","last_peered":"2020-11-09T12:37:23.057046+0000","last_clean":"2020-11-09T12:37:23.057046+0000","last_became_active":"2020-11-09T12:37:23.055097+0000","last_became_peered":"2020-11-09T12:37:23.055097+0000","last_unstale":"2020-11-09T12:37:23.057046+0000","last_undegraded":"2020-11-09T12:37:23.057046+0000","last_fullsized":"2020-11-09T12:37:23.057046+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:23.055270+0000","last_change":"2020-11-09T12:37:23.052973+0000","last_active":"2020-11-09T12:37:23.055270+0000","last_peered":"2020-11-09T12:37:23.055270+0000","last_clean":"2020-11-09T12:37:23.055270+0000","last_became_active":"2020-11-09T12:37:23.051846+0000","last_became_peered":"2020-11-09T12:37:23.051846+0000","last_unstale":"2020-11-09T12:37:23.055270+0000","last_undegraded":"2020-11-09T12:37:23.055270+0000","last_fullsized":"2020-11-09T12:37:23.055270+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:22.022485+0000","last_clean_scrub_stamp":"2020-11-09T12:37:22.022485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.373}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.603}]}]}]}} 2020-11-09T12:37:29.593 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-11-09T12:37:29.594 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-11-09T12:37:29.594 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2020-11-09T12:37:29.595 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2020-11-09T12:37:29.951 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:29.965 INFO:teuthology.orchestra.run.smithi164.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-09T12:37:29.966 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2020-11-09T12:37:29.966 INFO:teuthology.run_tasks:Running task ceph... 2020-11-09T12:37:30.072 INFO:tasks.ceph:Creating ceph cluster c2... 2020-11-09T12:37:30.073 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-09T12:37:30.073 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/ofriedma-2020-11-09_09:43:03-rgw-wip-ofriedma-prefetch-range-i21041-distro-basic-smithi/5605479', 'branch': 'wip-ofriedma-prefetch-range-i21041', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'ofriedma@redhat.com', 'first_in_suite': False, 'job_id': '5605479', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'ofriedma-2020-11-09_09:43:03-rgw-wip-ofriedma-prefetch-range-i21041-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-ofriedma-prefetch-range-i21041'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-ofriedma-prefetch-range-i21041', 'sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d'}}, 'owner': 'scheduled_ofriedma@teuthology', 'priority': 101, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-ofriedma-prefetch-range-i21041', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-ofriedma-prefetch-range-i21041/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'afece648f8ccac358b59a7b305a02d8c55dc908d', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2020-11-09_09:43:03', 'tube': 'smithi', 'user': 'ofriedma', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.11564', 'targets': {'smithi164.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDe4y9371xWFuJuIE04B1Fo1b3ZUH7GU9oATqw2xFl1QfUisADv12XzhKqc0ugtdpxyKvS/gawH3evS0UNqAISY3XCL6/tDGLyFGYGSKt64jJDB8sDZjAM70CW+2a4LHdnUOwU/1WgrN3UJt1bvxqBJx1JKxzf1tLXpbNS1ddAYdpFO8JfVRWDzAJE9A1sMDxwKLol7ugg8NcoXhgwIZy3c1njFGKsw2qLUDFlLX7uFl0ZcfXHSi/fbT75rz38IpYMnIBWsvJ7p6eeOZ2bggJEjqnxplzB53wy0tj0z++vsj1B7lAcm8poGBCEkckeoGJBs5USEpY/SJrhggEA4SUv5', 'smithi190.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQeVG89hdn6N416X1bJbSagXnM97175QAWzlHiMK4psAkbs3KVf9m0VKtccgsoREVvt7MGw3QTez9R2g4MBvQuto2PHUqUZqq5V4Pm6tBV4JnivVwTLThK/9yXXQDKyHaDV5tQ0q0KO9QtVod93+J+ksEm+zhq8BJM4l/gn08hIM/1adp4g8wOP1AjtZ+CdHm7PGUdDKTmrYpNx5Jw9dyGMKrTAcnq9sgWO7i1YlbXRIj1wTj3N6+X2DmoiI7jYLixC7D5IaSvwZ4BrnWqAKeIoG3Cbp2aQK01TLELHgAZnc9N0J/xD8rwZ2oLZTc8dXgzTkXqzStjS+v0erO1zVb5'}} 2020-11-09T12:37:30.074 INFO:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-11-09T12:37:30.077 INFO:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-11-09T12:37:30.084 INFO:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2020-11-09T12:37:30.126 INFO:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2020-11-09T12:37:30.141 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:30.142 INFO:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2020-11-09T12:37:30.188 INFO:teuthology.orchestra.run.smithi190.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-09T12:37:30.190 INFO:teuthology.orchestra.run.smithi190.stdout:/dev/vg_nvme/lv_1 2020-11-09T12:37:30.191 INFO:teuthology.orchestra.run.smithi190.stdout:/dev/vg_nvme/lv_2 2020-11-09T12:37:30.191 INFO:teuthology.orchestra.run.smithi190.stdout:/dev/vg_nvme/lv_3 2020-11-09T12:37:30.191 INFO:teuthology.orchestra.run.smithi190.stdout:/dev/vg_nvme/lv_4 2020-11-09T12:37:30.191 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:30.191 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:30.191 INFO:teuthology.orchestra.run.smithi190.stderr:72 bytes copied, 0.000168054 s, 428 kB/s 2020-11-09T12:37:30.192 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-09T12:37:30.192 INFO:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-09T12:37:30.239 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-09T12:37:30.239 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T12:37:30.240 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-11-09T12:37:30.240 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T12:37:30.240 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2020-11-09 12:34:42.418378823 +0000 2020-11-09T12:37:30.240 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2020-11-09 12:34:42.330334820 +0000 2020-11-09T12:37:30.240 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2020-11-09 12:34:42.330334820 +0000 2020-11-09T12:37:30.241 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2020-11-09T12:37:30.249 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2020-11-09T12:37:30.249 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2020-11-09T12:37:30.249 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000337211 s, 1.5 MB/s 2020-11-09T12:37:30.253 INFO:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-09T12:37:30.303 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-09T12:37:30.303 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T12:37:30.303 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-11-09T12:37:30.304 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T12:37:30.304 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2020-11-09 12:34:42.650494830 +0000 2020-11-09T12:37:30.304 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2020-11-09 12:34:42.562450828 +0000 2020-11-09T12:37:30.304 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2020-11-09 12:34:42.562450828 +0000 2020-11-09T12:37:30.304 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2020-11-09T12:37:30.313 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2020-11-09T12:37:30.314 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2020-11-09T12:37:30.314 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000391931 s, 1.3 MB/s 2020-11-09T12:37:30.317 INFO:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-09T12:37:30.366 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-09T12:37:30.367 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T12:37:30.367 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-11-09T12:37:30.367 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T12:37:30.367 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2020-11-09 12:34:42.858598837 +0000 2020-11-09T12:37:30.368 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2020-11-09 12:34:42.770554834 +0000 2020-11-09T12:37:30.368 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2020-11-09 12:34:42.770554834 +0000 2020-11-09T12:37:30.368 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2020-11-09T12:37:30.375 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2020-11-09T12:37:30.375 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2020-11-09T12:37:30.376 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000379694 s, 1.3 MB/s 2020-11-09T12:37:30.380 INFO:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-09T12:37:30.427 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-09T12:37:30.427 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-09T12:37:30.427 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-11-09T12:37:30.427 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-09T12:37:30.427 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2020-11-09 12:34:43.098718844 +0000 2020-11-09T12:37:30.428 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2020-11-09 12:34:43.010674842 +0000 2020-11-09T12:37:30.428 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2020-11-09 12:34:43.010674842 +0000 2020-11-09T12:37:30.428 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2020-11-09T12:37:30.435 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2020-11-09T12:37:30.436 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2020-11-09T12:37:30.436 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000382784 s, 1.3 MB/s 2020-11-09T12:37:30.439 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T12:37:30.439 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-09T12:37:30.439 INFO:tasks.ceph:Generating config... 2020-11-09T12:37:30.440 INFO:tasks.ceph:[client] debug rgw = 20 2020-11-09T12:37:30.441 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-11-09T12:37:30.441 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-11-09T12:37:30.441 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-09T12:37:30.441 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-11-09T12:37:30.442 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-11-09T12:37:30.442 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-11-09T12:37:30.442 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-11-09T12:37:30.442 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-11-09T12:37:30.443 INFO:tasks.ceph:[client] setgroup = ceph 2020-11-09T12:37:30.443 INFO:tasks.ceph:[client] setuser = ceph 2020-11-09T12:37:30.443 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-11-09T12:37:30.443 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-09T12:37:30.443 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-09T12:37:30.444 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-09T12:37:30.444 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-09T12:37:30.444 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-09T12:37:30.444 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-11-09T12:37:30.445 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-09T12:37:30.445 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-09T12:37:30.445 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-09T12:37:30.445 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-09T12:37:30.445 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-11-09T12:37:30.446 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-11-09T12:37:30.446 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-11-09T12:37:30.446 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-11-09T12:37:30.446 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-11-09T12:37:30.447 INFO:tasks.ceph:Setting up c2.mon.a... 2020-11-09T12:37:30.447 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-11-09T12:37:30.511 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/c2.keyring 2020-11-09T12:37:30.514 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-11-09T12:37:30.589 INFO:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-09T12:37:30.646 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.190')] 2020-11-09T12:37:30.647 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon host': '172.21.15.190', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-11-09T12:37:30.648 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:30.648 INFO:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-09T12:37:30.692 INFO:teuthology.orchestra.run.smithi190.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-09T12:37:30.694 INFO:teuthology.orchestra.run.smithi190.stderr:6+1 records in 2020-11-09T12:37:30.694 INFO:teuthology.orchestra.run.smithi190.stderr:6+1 records out 2020-11-09T12:37:30.694 INFO:teuthology.orchestra.run.smithi190.stderr:3101 bytes (3.1 kB, 3.0 KiB) copied, 0.000110421 s, 28.1 MB/s 2020-11-09T12:37:30.694 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.190 --print /home/ubuntu/cephtest/c2.monmap 2020-11-09T12:37:30.766 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:37:30.766 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:37:30.767 INFO:teuthology.orchestra.run.smithi190.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-11-09T12:37:30.767 INFO:teuthology.orchestra.run.smithi190.stdout:monmaptool: generated fsid 85b2f53f-98d9-4eb9-9d39-99874b789248 2020-11-09T12:37:30.767 INFO:teuthology.orchestra.run.smithi190.stdout:epoch 0 2020-11-09T12:37:30.768 INFO:teuthology.orchestra.run.smithi190.stdout:fsid 85b2f53f-98d9-4eb9-9d39-99874b789248 2020-11-09T12:37:30.768 INFO:teuthology.orchestra.run.smithi190.stdout:last_changed 2020-11-09T12:37:30.771233+0000 2020-11-09T12:37:30.768 INFO:teuthology.orchestra.run.smithi190.stdout:created 2020-11-09T12:37:30.771233+0000 2020-11-09T12:37:30.768 INFO:teuthology.orchestra.run.smithi190.stdout:min_mon_release 0 (unknown) 2020-11-09T12:37:30.768 INFO:teuthology.orchestra.run.smithi190.stdout:election_strategy: 1 2020-11-09T12:37:30.768 INFO:teuthology.orchestra.run.smithi190.stdout:0: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.a 2020-11-09T12:37:30.769 INFO:teuthology.orchestra.run.smithi190.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-11-09T12:37:30.770 INFO:teuthology.orchestra.run.smithi190:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-09T12:37:30.818 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 85b2f53f-98d9-4eb9-9d39-99874b789248... 2020-11-09T12:37:30.819 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-11-09T12:37:30.821 INFO:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-11-09T12:37:30.863 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2020-11-09T12:37:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2020-11-09T12:37:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-09T12:37:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2020-11-09T12:37:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2020-11-09T12:37:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2020-11-09T12:37:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2020-11-09T12:37:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2020-11-09T12:37:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2020-11-09T12:37:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2020-11-09T12:37:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-09T12:37:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2020-11-09T12:37:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2020-11-09T12:37:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-09T12:37:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2020-11-09T12:37:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2020-11-09T12:37:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2020-11-09T12:37:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2020-11-09T12:37:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2020-11-09T12:37:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2020-11-09T12:37:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2020-11-09T12:37:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2020-11-09T12:37:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-09T12:37:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-09T12:37:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2020-11-09T12:37:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-09T12:37:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2020-11-09T12:37:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2020-11-09T12:37:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2020-11-09T12:37:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2020-11-09T12:37:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = 172.21.15.190 2020-11-09T12:37:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: osd heartbeat grace = 40 2020-11-09T12:37:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 85b2f53f-98d9-4eb9-9d39-99874b789248 2020-11-09T12:37:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.870 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2020-11-09T12:37:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2020-11-09T12:37:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2020-11-09T12:37:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2020-11-09T12:37:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2020-11-09T12:37:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2020-11-09T12:37:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2020-11-09T12:37:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2020-11-09T12:37:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2020-11-09T12:37:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2020-11-09T12:37:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2020-11-09T12:37:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2020-11-09T12:37:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2020-11-09T12:37:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2020-11-09T12:37:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2020-11-09T12:37:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2020-11-09T12:37:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.875 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2020-11-09T12:37:30.875 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2020-11-09T12:37:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: debug filestore = 20 2020-11-09T12:37:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: debug journal = 20 2020-11-09T12:37:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 20 2020-11-09T12:37:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 25 2020-11-09T12:37:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: osd fast shutdown = False 2020-11-09T12:37:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: rocksdb delete range threshold = 0 2020-11-09T12:37:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.877 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2020-11-09T12:37:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2020-11-09T12:37:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2020-11-09T12:37:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2020-11-09T12:37:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2020-11-09T12:37:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2020-11-09T12:37:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2020-11-09T12:37:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2020-11-09T12:37:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.878 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2020-11-09T12:37:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2020-11-09T12:37:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2020-11-09T12:37:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2020-11-09T12:37:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2020-11-09T12:37:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2020-11-09T12:37:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2020-11-09T12:37:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2020-11-09T12:37:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2020-11-09T12:37:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2020-11-09T12:37:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2020-11-09T12:37:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd crush smoke test = False 2020-11-09T12:37:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:37:30.880 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2020-11-09T12:37:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2020-11-09T12:37:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2020-11-09T12:37:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2020-11-09T12:37:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-09T12:37:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-09T12:37:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: debug rgw = 20 2020-11-09T12:37:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: rgw crypt require ssl = False 2020-11-09T12:37:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: rgw crypt s3 kms backend = testing 2020-11-09T12:37:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-09T12:37:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: rgw curl low speed time = 300 2020-11-09T12:37:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: rgw data log num shards = 4 2020-11-09T12:37:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: rgw md log max shards = 4 2020-11-09T12:37:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: rgw sync log trim interval = 0 2020-11-09T12:37:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: rgw sync obj etag verify = True 2020-11-09T12:37:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: setgroup = ceph 2020-11-09T12:37:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: setuser = ceph 2020-11-09T12:37:30.882 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2020-11-09T12:37:30.882 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.0] 2020-11-09T12:37:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: osd_max_omap_entries_per_request = 10 2020-11-09T12:37:30.882 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.1] 2020-11-09T12:37:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: osd_max_omap_entries_per_request = 1000 2020-11-09T12:37:30.883 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.2] 2020-11-09T12:37:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: osd_max_omap_entries_per_request = 10000 2020-11-09T12:37:30.891 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2020-11-09T12:37:30.892 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2020-11-09T12:37:30.892 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-09T12:37:30.892 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2020-11-09T12:37:30.892 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.892 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2020-11-09T12:37:30.893 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.893 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2020-11-09T12:37:30.893 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.893 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2020-11-09T12:37:30.893 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2020-11-09T12:37:30.893 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.893 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2020-11-09T12:37:30.894 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2020-11-09T12:37:30.894 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.894 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-09T12:37:30.894 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2020-11-09T12:37:30.894 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.894 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2020-11-09T12:37:30.894 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-09T12:37:30.895 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.895 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2020-11-09T12:37:30.895 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.896 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2020-11-09T12:37:30.896 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2020-11-09T12:37:30.896 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2020-11-09T12:37:30.896 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2020-11-09T12:37:30.896 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2020-11-09T12:37:30.896 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2020-11-09T12:37:30.896 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2020-11-09T12:37:30.897 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-09T12:37:30.897 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-09T12:37:30.897 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2020-11-09T12:37:30.897 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.897 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-09T12:37:30.897 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.897 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2020-11-09T12:37:30.897 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.898 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2020-11-09T12:37:30.898 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.898 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2020-11-09T12:37:30.898 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2020-11-09T12:37:30.898 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = 172.21.15.190 2020-11-09T12:37:30.898 INFO:teuthology.orchestra.run.smithi190.stdout: osd heartbeat grace = 40 2020-11-09T12:37:30.898 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = 85b2f53f-98d9-4eb9-9d39-99874b789248 2020-11-09T12:37:30.899 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.899 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2020-11-09T12:37:30.899 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2020-11-09T12:37:30.899 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.899 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2020-11-09T12:37:30.899 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 600 2020-11-09T12:37:30.899 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.899 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2020-11-09T12:37:30.900 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2020-11-09T12:37:30.900 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.900 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2020-11-09T12:37:30.900 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2020-11-09T12:37:30.900 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2020-11-09T12:37:30.900 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.900 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2020-11-09T12:37:30.901 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2020-11-09T12:37:30.901 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.901 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2020-11-09T12:37:30.901 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.901 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2020-11-09T12:37:30.901 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.901 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2020-11-09T12:37:30.901 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.902 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2020-11-09T12:37:30.902 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2020-11-09T12:37:30.902 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.902 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2020-11-09T12:37:30.902 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2020-11-09T12:37:30.902 INFO:teuthology.orchestra.run.smithi190.stdout: debug filestore = 20 2020-11-09T12:37:30.902 INFO:teuthology.orchestra.run.smithi190.stdout: debug journal = 20 2020-11-09T12:37:30.903 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 20 2020-11-09T12:37:30.903 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 25 2020-11-09T12:37:30.903 INFO:teuthology.orchestra.run.smithi190.stdout: osd fast shutdown = False 2020-11-09T12:37:30.903 INFO:teuthology.orchestra.run.smithi190.stdout: rocksdb delete range threshold = 0 2020-11-09T12:37:30.903 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.903 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2020-11-09T12:37:30.903 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2020-11-09T12:37:30.904 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2020-11-09T12:37:30.904 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2020-11-09T12:37:30.904 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2020-11-09T12:37:30.904 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2020-11-09T12:37:30.904 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2020-11-09T12:37:30.904 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2020-11-09T12:37:30.904 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.904 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2020-11-09T12:37:30.905 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2020-11-09T12:37:30.905 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2020-11-09T12:37:30.905 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2020-11-09T12:37:30.905 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2020-11-09T12:37:30.905 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2020-11-09T12:37:30.906 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2020-11-09T12:37:30.906 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2020-11-09T12:37:30.906 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2020-11-09T12:37:30.906 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2020-11-09T12:37:30.906 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2020-11-09T12:37:30.907 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd crush smoke test = False 2020-11-09T12:37:30.907 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:30.907 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2020-11-09T12:37:30.907 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2020-11-09T12:37:30.907 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2020-11-09T12:37:30.907 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2020-11-09T12:37:30.907 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-09T12:37:30.908 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-09T12:37:30.908 INFO:teuthology.orchestra.run.smithi190.stdout: debug rgw = 20 2020-11-09T12:37:30.908 INFO:teuthology.orchestra.run.smithi190.stdout: rgw crypt require ssl = False 2020-11-09T12:37:30.908 INFO:teuthology.orchestra.run.smithi190.stdout: rgw crypt s3 kms backend = testing 2020-11-09T12:37:30.908 INFO:teuthology.orchestra.run.smithi190.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-09T12:37:30.908 INFO:teuthology.orchestra.run.smithi190.stdout: rgw curl low speed time = 300 2020-11-09T12:37:30.908 INFO:teuthology.orchestra.run.smithi190.stdout: rgw data log num shards = 4 2020-11-09T12:37:30.909 INFO:teuthology.orchestra.run.smithi190.stdout: rgw md log max shards = 4 2020-11-09T12:37:30.909 INFO:teuthology.orchestra.run.smithi190.stdout: rgw sync log trim interval = 0 2020-11-09T12:37:30.909 INFO:teuthology.orchestra.run.smithi190.stdout: rgw sync obj etag verify = True 2020-11-09T12:37:30.909 INFO:teuthology.orchestra.run.smithi190.stdout: setgroup = ceph 2020-11-09T12:37:30.909 INFO:teuthology.orchestra.run.smithi190.stdout: setuser = ceph 2020-11-09T12:37:30.909 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2020-11-09T12:37:30.909 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.0] 2020-11-09T12:37:30.910 INFO:teuthology.orchestra.run.smithi190.stdout: osd_max_omap_entries_per_request = 10 2020-11-09T12:37:30.910 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.1] 2020-11-09T12:37:30.910 INFO:teuthology.orchestra.run.smithi190.stdout: osd_max_omap_entries_per_request = 1000 2020-11-09T12:37:30.910 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.2] 2020-11-09T12:37:30.910 INFO:teuthology.orchestra.run.smithi190.stdout: osd_max_omap_entries_per_request = 10000 2020-11-09T12:37:30.911 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-11-09T12:37:30.912 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-11-09T12:37:30.996 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-09T12:37:30.997 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:30.997 INFO:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-11-09T12:37:31.044 INFO:teuthology.orchestra.run.smithi190.stderr:+ dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-11-09T12:37:31.046 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.] 2020-11-09T12:37:31.046 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAKOKlfO6dEIxAA5O/aakq5s0s/OLs2fpaE/Q== 2020-11-09T12:37:31.046 INFO:teuthology.orchestra.run.smithi190.stdout:[client.admin] 2020-11-09T12:37:31.046 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAKOKlfhC+UOxAAoNQkNRWPhW57LtZRT2+ukA== 2020-11-09T12:37:31.047 INFO:teuthology.orchestra.run.smithi190.stdout: caps mds = "allow *" 2020-11-09T12:37:31.047 INFO:teuthology.orchestra.run.smithi190.stdout: caps mgr = "allow *" 2020-11-09T12:37:31.047 INFO:teuthology.orchestra.run.smithi190.stdout: caps mon = "allow *" 2020-11-09T12:37:31.047 INFO:teuthology.orchestra.run.smithi190.stdout: caps osd = "allow *" 2020-11-09T12:37:31.047 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:31.047 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:31.047 INFO:teuthology.orchestra.run.smithi190.stderr:206 bytes copied, 0.000173576 s, 1.2 MB/s 2020-11-09T12:37:31.048 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:31.048 INFO:teuthology.orchestra.run.smithi190:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-11-09T12:37:31.092 INFO:teuthology.orchestra.run.smithi190.stderr:+ dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-11-09T12:37:31.093 INFO:teuthology.orchestra.run.smithi190.stdout: ����?��N��9��Kx�H 2020-11-09T12:37:31.094 INFO:teuthology.orchestra.run.smithi190.stdout:8�_��- 2020-11-09T12:37:31.094 INFO:teuthology.orchestra.run.smithi190.stdout:8�_��-�aXa �����a 2020-11-09T12:37:31.094 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:31.094 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:31.094 INFO:teuthology.orchestra.run.smithi190.stderr:201 bytes copied, 0.000156501 s, 1.3 MB/s 2020-11-09T12:37:31.095 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2020-11-09T12:37:31.095 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:31.095 INFO:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/c2.keyring 2020-11-09T12:37:31.095 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-09T12:37:31.100 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-11-09T12:37:31.110 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:31.110 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:31.110 INFO:teuthology.orchestra.run.smithi164.stderr:206 bytes copied, 6.7453e-05 s, 3.1 MB/s 2020-11-09T12:37:31.111 INFO:teuthology.orchestra.run.smithi164.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-09T12:37:31.121 INFO:teuthology.orchestra.run.smithi164:> set -ex 2020-11-09T12:37:31.121 INFO:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-09T12:37:31.169 INFO:teuthology.orchestra.run.smithi164.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-09T12:37:31.170 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records in 2020-11-09T12:37:31.170 INFO:teuthology.orchestra.run.smithi164.stderr:0+1 records out 2020-11-09T12:37:31.171 INFO:teuthology.orchestra.run.smithi164.stderr:201 bytes copied, 6.961e-05 s, 2.9 MB/s 2020-11-09T12:37:31.171 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2020-11-09T12:37:31.171 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:31.171 INFO:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/c2.keyring 2020-11-09T12:37:31.172 INFO:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-09T12:37:31.176 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-11-09T12:37:31.185 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:31.186 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:31.186 INFO:teuthology.orchestra.run.smithi190.stderr:206 bytes copied, 0.00012934 s, 1.6 MB/s 2020-11-09T12:37:31.187 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-09T12:37:31.195 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:31.195 INFO:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-09T12:37:31.244 INFO:teuthology.orchestra.run.smithi190.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-09T12:37:31.246 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:31.246 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:31.247 INFO:teuthology.orchestra.run.smithi190.stderr:201 bytes copied, 0.000133324 s, 1.5 MB/s 2020-11-09T12:37:31.247 INFO:tasks.ceph:Setting up mon nodes... 2020-11-09T12:37:31.247 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-09T12:37:31.247 INFO:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-11-09T12:37:31.329 INFO:teuthology.orchestra.run.smithi190.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-11-09T12:37:31.331 INFO:tasks.ceph:Setting up mds nodes... 2020-11-09T12:37:31.331 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-09T12:37:31.331 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-11-09T12:37:31.404 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/c2.client.0.keyring 2020-11-09T12:37:31.416 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-11-09T12:37:31.492 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/c2.client.1.keyring 2020-11-09T12:37:31.505 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-09T12:37:31.505 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-09T12:37:31.505 INFO:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-11-09T12:37:31.563 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T12:37:31.563 INFO:tasks.ceph:role: c2.osd.0 2020-11-09T12:37:31.563 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com 2020-11-09T12:37:31.564 INFO:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-09T12:37:32.080 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-09T12:37:32.080 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-09T12:37:32.080 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-09T12:37:32.080 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-09T12:37:32.081 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2020-11-09T12:37:32.081 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-09T12:37:32.081 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-09T12:37:32.081 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-09T12:37:32.081 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-09T12:37:32.082 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2020-11-09T12:37:32.083 INFO:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2020-11-09T12:37:32.159 INFO:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-11-09T12:37:32.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2020-11-09T12:37:32.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-09T12:37:32.173 INFO:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-11-09T12:37:32.231 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T12:37:32.232 INFO:tasks.ceph:role: c2.osd.1 2020-11-09T12:37:32.232 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com 2020-11-09T12:37:32.232 INFO:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-09T12:37:32.742 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-09T12:37:32.743 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-09T12:37:32.743 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-09T12:37:32.743 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-09T12:37:32.743 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2020-11-09T12:37:32.743 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-09T12:37:32.743 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-09T12:37:32.743 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-09T12:37:32.744 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-09T12:37:32.744 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2020-11-09T12:37:32.745 INFO:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2020-11-09T12:37:32.811 INFO:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-11-09T12:37:32.821 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2020-11-09T12:37:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-09T12:37:32.823 INFO:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-11-09T12:37:32.878 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-09T12:37:32.879 INFO:tasks.ceph:role: c2.osd.2 2020-11-09T12:37:32.879 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com 2020-11-09T12:37:32.879 INFO:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-09T12:37:33.394 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-09T12:37:33.395 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-09T12:37:33.395 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-09T12:37:33.395 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-09T12:37:33.395 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2020-11-09T12:37:33.395 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-09T12:37:33.395 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-09T12:37:33.395 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-09T12:37:33.396 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-09T12:37:33.396 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2020-11-09T12:37:33.397 INFO:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2020-11-09T12:37:33.450 INFO:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-11-09T12:37:33.461 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2020-11-09T12:37:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-09T12:37:33.462 INFO:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-09T12:37:33.554 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:33.556+0000 7fa753605d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-11-09T12:37:33.555 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:33.556+0000 7fa753605d80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-11-09T12:37:33.555 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:33.556+0000 7fa753605d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-09T12:37:33.555 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:33.556+0000 7fa753605d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-09T12:37:33.556 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:33.556+0000 7fa753605d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-09T12:37:33.563 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:33.564+0000 7fa753605d80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-11-09T12:37:35.747 INFO:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-11-09T12:37:35.762 INFO:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-09T12:37:35.842 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:35.844+0000 7f11c6390d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-11-09T12:37:35.842 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:35.844+0000 7f11c6390d80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-11-09T12:37:35.843 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:35.844+0000 7f11c6390d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-09T12:37:35.843 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:35.844+0000 7f11c6390d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-09T12:37:35.843 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:35.844+0000 7f11c6390d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-09T12:37:35.843 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:35.844+0000 7f11c6390d80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-11-09T12:37:38.028 INFO:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-11-09T12:37:38.087 INFO:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-09T12:37:38.165 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:38.168+0000 7fa54956dd80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-11-09T12:37:38.166 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:38.168+0000 7fa54956dd80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-11-09T12:37:38.166 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:38.168+0000 7fa54956dd80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-09T12:37:38.166 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:38.168+0000 7fa54956dd80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-09T12:37:38.166 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:38.168+0000 7fa54956dd80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-09T12:37:38.167 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:37:38.168+0000 7fa54956dd80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-11-09T12:37:40.330 INFO:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-11-09T12:37:40.391 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-09T12:37:40.392 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:40.392 INFO:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-11-09T12:37:40.436 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-11-09T12:37:40.445 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.x] 2020-11-09T12:37:40.445 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQALOKlfvTbkExAAbnpkIiKC0MHmc2+jMy1dDQ== 2020-11-09T12:37:40.445 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:40.445 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:40.446 INFO:teuthology.orchestra.run.smithi190.stderr:56 bytes copied, 0.000183826 s, 305 kB/s 2020-11-09T12:37:40.446 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:40.447 INFO:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-11-09T12:37:40.492 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-11-09T12:37:40.500 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.0] 2020-11-09T12:37:40.500 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQANOKlf0xRHIRAA5Y8SnYEousOSPl6Dzc7V7g== 2020-11-09T12:37:40.501 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:40.501 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:40.501 INFO:teuthology.orchestra.run.smithi190.stderr:56 bytes copied, 0.000189207 s, 296 kB/s 2020-11-09T12:37:40.502 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:40.502 INFO:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-11-09T12:37:40.554 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-11-09T12:37:40.563 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.1] 2020-11-09T12:37:40.563 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAPOKlfkfV7MhAASKA/cvuNxsHNWUxKplaOIQ== 2020-11-09T12:37:40.563 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:40.563 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:40.563 INFO:teuthology.orchestra.run.smithi190.stderr:56 bytes copied, 0.000169497 s, 330 kB/s 2020-11-09T12:37:40.564 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:40.565 INFO:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-11-09T12:37:40.612 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-11-09T12:37:40.620 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.2] 2020-11-09T12:37:40.621 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQASOKlfHrklChAAABVOXISkqEMy1iNIC2U0bg== 2020-11-09T12:37:40.621 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:40.621 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:40.621 INFO:teuthology.orchestra.run.smithi190.stderr:56 bytes copied, 0.000175371 s, 319 kB/s 2020-11-09T12:37:40.622 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:40.622 INFO:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-11-09T12:37:40.668 INFO:teuthology.orchestra.run.smithi190.stderr:+ dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-11-09T12:37:40.669 INFO:teuthology.orchestra.run.smithi190.stdout:[client.0] 2020-11-09T12:37:40.669 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQALOKlfutZoGBAARS6mvdbycZlmH+P1a3sBeQ== 2020-11-09T12:37:40.670 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:40.670 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:40.670 INFO:teuthology.orchestra.run.smithi190.stderr:59 bytes copied, 0.000157803 s, 374 kB/s 2020-11-09T12:37:40.670 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:40.670 INFO:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-11-09T12:37:40.716 INFO:teuthology.orchestra.run.smithi190.stderr:+ dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-11-09T12:37:40.717 INFO:teuthology.orchestra.run.smithi190.stdout:[client.1] 2020-11-09T12:37:40.717 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQALOKlfBkSnHRAAF6JMRjXvakCv1mB4So9eUg== 2020-11-09T12:37:40.718 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:40.718 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:40.718 INFO:teuthology.orchestra.run.smithi190.stderr:59 bytes copied, 0.000165177 s, 357 kB/s 2020-11-09T12:37:40.718 INFO:tasks.ceph:Adding keys to all mons... 2020-11-09T12:37:40.718 INFO:teuthology.orchestra.run.smithi190:> sudo tee -a /etc/ceph/c2.keyring 2020-11-09T12:37:40.772 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.x] 2020-11-09T12:37:40.773 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQALOKlfvTbkExAAbnpkIiKC0MHmc2+jMy1dDQ== 2020-11-09T12:37:40.773 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.0] 2020-11-09T12:37:40.773 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQANOKlf0xRHIRAA5Y8SnYEousOSPl6Dzc7V7g== 2020-11-09T12:37:40.773 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.1] 2020-11-09T12:37:40.773 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAPOKlfkfV7MhAASKA/cvuNxsHNWUxKplaOIQ== 2020-11-09T12:37:40.773 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.2] 2020-11-09T12:37:40.774 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQASOKlfHrklChAAABVOXISkqEMy1iNIC2U0bg== 2020-11-09T12:37:40.774 INFO:teuthology.orchestra.run.smithi190.stdout:[client.0] 2020-11-09T12:37:40.774 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQALOKlfutZoGBAARS6mvdbycZlmH+P1a3sBeQ== 2020-11-09T12:37:40.774 INFO:teuthology.orchestra.run.smithi190.stdout:[client.1] 2020-11-09T12:37:40.774 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQALOKlfBkSnHRAAF6JMRjXvakCv1mB4So9eUg== 2020-11-09T12:37:40.775 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-09T12:37:40.860 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-09T12:37:40.945 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-09T12:37:41.030 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-09T12:37:41.114 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-09T12:37:41.200 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-09T12:37:41.288 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-09T12:37:41.289 INFO:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-11-09T12:37:41.347 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-11-09T12:37:41.442 INFO:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-11-09T12:37:41.453 INFO:teuthology.orchestra.run.smithi190:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-11-09T12:37:41.501 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-11-09T12:37:41.502 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-11-09T12:37:41.502 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2020-11-09T12:37:41.545 INFO:tasks.ceph.c2.mon.a:Started 2020-11-09T12:37:41.546 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-11-09T12:37:41.546 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-11-09T12:37:41.547 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2020-11-09T12:37:41.549 INFO:tasks.ceph.c2.mgr.x:Started 2020-11-09T12:37:41.549 INFO:tasks.ceph:Setting crush tunables to default 2020-11-09T12:37:41.549 INFO:teuthology.orchestra.run.smithi190:> sudo ceph --cluster c2 osd crush tunables default 2020-11-09T12:37:41.868 INFO:teuthology.orchestra.run.smithi190.stderr:adjusted tunables profile to default 2020-11-09T12:37:41.880 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-11-09T12:37:41.880 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:41.880 INFO:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-11-09T12:37:41.885 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-11-09T12:37:41.893 INFO:teuthology.orchestra.run.smithi190.stdout:7a7f97f9-f92b-4d6e-8086-be39e96e43d1 2020-11-09T12:37:41.893 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:41.893 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:41.893 INFO:teuthology.orchestra.run.smithi190.stderr:37 bytes copied, 0.000117357 s, 315 kB/s 2020-11-09T12:37:41.894 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:41.895 INFO:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-11-09T12:37:41.940 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-11-09T12:37:41.947 INFO:teuthology.orchestra.run.smithi190.stdout:bc0af409-4279-46e6-af45-b0eab50c353e 2020-11-09T12:37:41.947 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:41.947 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:41.947 INFO:teuthology.orchestra.run.smithi190.stderr:37 bytes copied, 0.00022084 s, 168 kB/s 2020-11-09T12:37:41.948 INFO:teuthology.orchestra.run.smithi190:> set -ex 2020-11-09T12:37:41.949 INFO:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-11-09T12:37:41.996 INFO:teuthology.orchestra.run.smithi190.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-11-09T12:37:42.004 INFO:teuthology.orchestra.run.smithi190.stdout:03a55901-9fae-41f9-9b37-1cf54c976044 2020-11-09T12:37:42.004 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records in 2020-11-09T12:37:42.004 INFO:teuthology.orchestra.run.smithi190.stderr:0+1 records out 2020-11-09T12:37:42.005 INFO:teuthology.orchestra.run.smithi190.stderr:37 bytes copied, 0.00023077 s, 160 kB/s 2020-11-09T12:37:42.005 INFO:teuthology.orchestra.run.smithi190:> sudo ceph --cluster c2 osd new 7a7f97f9-f92b-4d6e-8086-be39e96e43d1 0 2020-11-09T12:37:42.263 INFO:teuthology.orchestra.run.smithi190.stdout:0 2020-11-09T12:37:42.276 INFO:teuthology.orchestra.run.smithi190:> sudo ceph --cluster c2 osd new bc0af409-4279-46e6-af45-b0eab50c353e 1 2020-11-09T12:37:42.507 INFO:teuthology.orchestra.run.smithi190.stdout:1 2020-11-09T12:37:42.519 INFO:teuthology.orchestra.run.smithi190:> sudo ceph --cluster c2 osd new 03a55901-9fae-41f9-9b37-1cf54c976044 2 2020-11-09T12:37:42.725 INFO:teuthology.orchestra.run.smithi190.stdout:2 2020-11-09T12:37:42.739 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-11-09T12:37:42.739 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2020-11-09T12:37:42.741 INFO:tasks.ceph.c2.osd.0:Started 2020-11-09T12:37:42.741 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-11-09T12:37:42.742 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2020-11-09T12:37:42.744 INFO:tasks.ceph.c2.osd.1:Started 2020-11-09T12:37:42.744 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-11-09T12:37:42.744 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2020-11-09T12:37:42.746 INFO:tasks.ceph.c2.osd.2:Started 2020-11-09T12:37:42.747 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-09T12:37:43.049 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-11-09T12:37:43.089 INFO:tasks.ceph.c2.osd.0.smithi190.stderr:2020-11-09T12:37:43.092+0000 7fe4eab82d80 -1 Falling back to public interface 2020-11-09T12:37:43.237 INFO:teuthology.misc.health.smithi190.stdout: 2020-11-09T12:37:43.237 INFO:teuthology.misc.health.smithi190.stdout:{"epoch":5,"fsid":"85b2f53f-98d9-4eb9-9d39-99874b789248","created":"2020-11-09T12:37:41.672623+0000","modified":"2020-11-09T12:37:42.727951+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"7a7f97f9-f92b-4d6e-8086-be39e96e43d1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bc0af409-4279-46e6-af45-b0eab50c353e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03a55901-9fae-41f9-9b37-1cf54c976044","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T12:37:43.248 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-11-09T12:37:43.350 INFO:tasks.ceph.c2.osd.1.smithi190.stderr:2020-11-09T12:37:43.352+0000 7fd672de6d80 -1 Falling back to public interface 2020-11-09T12:37:43.353 INFO:tasks.ceph.c2.osd.2.smithi190.stderr:2020-11-09T12:37:43.356+0000 7fa4d2228d80 -1 Falling back to public interface 2020-11-09T12:37:43.727 INFO:tasks.ceph.c2.osd.1.smithi190.stderr:2020-11-09T12:37:43.728+0000 7fd672de6d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-09T12:37:43.731 INFO:tasks.ceph.c2.osd.0.smithi190.stderr:2020-11-09T12:37:43.732+0000 7fe4eab82d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-09T12:37:43.756 INFO:tasks.ceph.c2.osd.2.smithi190.stderr:2020-11-09T12:37:43.760+0000 7fa4d2228d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-09T12:37:45.696 INFO:tasks.ceph.c2.osd.0.smithi190.stderr:2020-11-09T12:37:45.696+0000 7fe4e204c700 -1 osd.0 0 waiting for initial osdmap 2020-11-09T12:37:45.697 INFO:tasks.ceph.c2.osd.1.smithi190.stderr:2020-11-09T12:37:45.700+0000 7fd66a2b0700 -1 osd.1 0 waiting for initial osdmap 2020-11-09T12:37:45.697 INFO:tasks.ceph.c2.osd.2.smithi190.stderr:2020-11-09T12:37:45.700+0000 7fa4caef5700 -1 osd.2 0 waiting for initial osdmap 2020-11-09T12:37:49.552 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-11-09T12:37:49.841 INFO:teuthology.misc.health.smithi190.stdout: 2020-11-09T12:37:49.842 INFO:teuthology.misc.health.smithi190.stdout:{"epoch":10,"fsid":"85b2f53f-98d9-4eb9-9d39-99874b789248","created":"2020-11-09T12:37:41.672623+0000","modified":"2020-11-09T12:37:48.717774+0000","last_up_change":"2020-11-09T12:37:46.696522+0000","last_in_change":"2020-11-09T12:37:46.696522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T12:37:45.570105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"7a7f97f9-f92b-4d6e-8086-be39e96e43d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":12414},{"type":"v1","addr":"172.21.15.190:6801","nonce":12414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":12414},{"type":"v1","addr":"172.21.15.190:6803","nonce":12414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":12414},{"type":"v1","addr":"172.21.15.190:6807","nonce":12414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":12414},{"type":"v1","addr":"172.21.15.190:6805","nonce":12414}]},"public_addr":"172.21.15.190:6801/12414","cluster_addr":"172.21.15.190:6803/12414","heartbeat_back_addr":"172.21.15.190:6807/12414","heartbeat_front_addr":"172.21.15.190:6805/12414","state":["exists","up"]},{"osd":1,"uuid":"bc0af409-4279-46e6-af45-b0eab50c353e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":12416},{"type":"v1","addr":"172.21.15.190:6809","nonce":12416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":12416},{"type":"v1","addr":"172.21.15.190:6811","nonce":12416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":12416},{"type":"v1","addr":"172.21.15.190:6816","nonce":12416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":12416},{"type":"v1","addr":"172.21.15.190:6813","nonce":12416}]},"public_addr":"172.21.15.190:6809/12416","cluster_addr":"172.21.15.190:6811/12416","heartbeat_back_addr":"172.21.15.190:6816/12416","heartbeat_front_addr":"172.21.15.190:6813/12416","state":["exists","up"]},{"osd":2,"uuid":"03a55901-9fae-41f9-9b37-1cf54c976044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":12415},{"type":"v1","addr":"172.21.15.190:6817","nonce":12415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":12415},{"type":"v1","addr":"172.21.15.190:6819","nonce":12415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":12415},{"type":"v1","addr":"172.21.15.190:6823","nonce":12415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":12415},{"type":"v1","addr":"172.21.15.190:6821","nonce":12415}]},"public_addr":"172.21.15.190:6817/12415","cluster_addr":"172.21.15.190:6819/12415","heartbeat_back_addr":"172.21.15.190:6823/12415","heartbeat_front_addr":"172.21.15.190:6821/12415","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:44.761435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:44.752911+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:44.784588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T12:37:49.854 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-11-09T12:37:49.854 INFO:tasks.ceph:Creating RBD pool 2020-11-09T12:37:49.855 INFO:teuthology.orchestra.run.smithi190:> sudo ceph --cluster c2 osd pool create rbd 8 2020-11-09T12:37:50.752 INFO:teuthology.orchestra.run.smithi190.stderr:pool 'rbd' created 2020-11-09T12:37:50.765 INFO:teuthology.orchestra.run.smithi190:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-09T12:37:51.766 INFO:teuthology.orchestra.run.smithi190.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-09T12:37:51.778 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-11-09T12:37:51.778 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-09T12:37:51.779 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-09T12:37:52.057 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:52.057 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":12,"fsid":"85b2f53f-98d9-4eb9-9d39-99874b789248","created":"2020-11-09T12:37:41.672623+0000","modified":"2020-11-09T12:37:51.744050+0000","last_up_change":"2020-11-09T12:37:46.696522+0000","last_in_change":"2020-11-09T12:37:46.696522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T12:37:45.570105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-09T12:37:50.150053+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7a7f97f9-f92b-4d6e-8086-be39e96e43d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":12414},{"type":"v1","addr":"172.21.15.190:6801","nonce":12414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":12414},{"type":"v1","addr":"172.21.15.190:6803","nonce":12414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":12414},{"type":"v1","addr":"172.21.15.190:6807","nonce":12414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":12414},{"type":"v1","addr":"172.21.15.190:6805","nonce":12414}]},"public_addr":"172.21.15.190:6801/12414","cluster_addr":"172.21.15.190:6803/12414","heartbeat_back_addr":"172.21.15.190:6807/12414","heartbeat_front_addr":"172.21.15.190:6805/12414","state":["exists","up"]},{"osd":1,"uuid":"bc0af409-4279-46e6-af45-b0eab50c353e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":12416},{"type":"v1","addr":"172.21.15.190:6809","nonce":12416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":12416},{"type":"v1","addr":"172.21.15.190:6811","nonce":12416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":12416},{"type":"v1","addr":"172.21.15.190:6816","nonce":12416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":12416},{"type":"v1","addr":"172.21.15.190:6813","nonce":12416}]},"public_addr":"172.21.15.190:6809/12416","cluster_addr":"172.21.15.190:6811/12416","heartbeat_back_addr":"172.21.15.190:6816/12416","heartbeat_front_addr":"172.21.15.190:6813/12416","state":["exists","up"]},{"osd":2,"uuid":"03a55901-9fae-41f9-9b37-1cf54c976044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":12415},{"type":"v1","addr":"172.21.15.190:6817","nonce":12415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":12415},{"type":"v1","addr":"172.21.15.190:6819","nonce":12415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":12415},{"type":"v1","addr":"172.21.15.190:6823","nonce":12415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":12415},{"type":"v1","addr":"172.21.15.190:6821","nonce":12415}]},"public_addr":"172.21.15.190:6817/12415","cluster_addr":"172.21.15.190:6819/12415","heartbeat_back_addr":"172.21.15.190:6823/12415","heartbeat_front_addr":"172.21.15.190:6821/12415","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:44.761435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:44.752911+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:44.784588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T12:37:52.071 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-09T12:37:45.570105+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-09T12:37:50.150053+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-09T12:37:52.072 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get device_health_metrics pg_num 2020-11-09T12:37:52.362 INFO:teuthology.orchestra.run.smithi190.stdout:pg_num: 1 2020-11-09T12:37:52.384 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2020-11-09T12:37:52.667 INFO:teuthology.orchestra.run.smithi190.stdout:pg_num: 8 2020-11-09T12:37:52.681 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-11-09T12:37:52.681 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-11-09T12:37:52.682 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2020-11-09T12:37:53.002 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:53.015 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":12231},{"type":"v1","addr":"172.21.15.190:6825","nonce":12231}]},"active_addr":"172.21.15.190:6825/12231","active_change":"2020-11-09T12:37:45.531172+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":1427831040}]},{"addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":3544259656}]},{"addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":1571676553}]}]}} 2020-11-09T12:37:53.017 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-11-09T12:37:53.018 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2020-11-09T12:37:53.018 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-09T12:37:53.308 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:53.308 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":13,"fsid":"85b2f53f-98d9-4eb9-9d39-99874b789248","created":"2020-11-09T12:37:41.672623+0000","modified":"2020-11-09T12:37:52.753068+0000","last_up_change":"2020-11-09T12:37:46.696522+0000","last_in_change":"2020-11-09T12:37:46.696522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T12:37:45.570105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-09T12:37:50.150053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7a7f97f9-f92b-4d6e-8086-be39e96e43d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":12414},{"type":"v1","addr":"172.21.15.190:6801","nonce":12414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":12414},{"type":"v1","addr":"172.21.15.190:6803","nonce":12414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":12414},{"type":"v1","addr":"172.21.15.190:6807","nonce":12414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":12414},{"type":"v1","addr":"172.21.15.190:6805","nonce":12414}]},"public_addr":"172.21.15.190:6801/12414","cluster_addr":"172.21.15.190:6803/12414","heartbeat_back_addr":"172.21.15.190:6807/12414","heartbeat_front_addr":"172.21.15.190:6805/12414","state":["exists","up"]},{"osd":1,"uuid":"bc0af409-4279-46e6-af45-b0eab50c353e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":12416},{"type":"v1","addr":"172.21.15.190:6809","nonce":12416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":12416},{"type":"v1","addr":"172.21.15.190:6811","nonce":12416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":12416},{"type":"v1","addr":"172.21.15.190:6816","nonce":12416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":12416},{"type":"v1","addr":"172.21.15.190:6813","nonce":12416}]},"public_addr":"172.21.15.190:6809/12416","cluster_addr":"172.21.15.190:6811/12416","heartbeat_back_addr":"172.21.15.190:6816/12416","heartbeat_front_addr":"172.21.15.190:6813/12416","state":["exists","up"]},{"osd":2,"uuid":"03a55901-9fae-41f9-9b37-1cf54c976044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":12415},{"type":"v1","addr":"172.21.15.190:6817","nonce":12415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":12415},{"type":"v1","addr":"172.21.15.190:6819","nonce":12415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":12415},{"type":"v1","addr":"172.21.15.190:6823","nonce":12415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":12415},{"type":"v1","addr":"172.21.15.190:6821","nonce":12415}]},"public_addr":"172.21.15.190:6817/12415","cluster_addr":"172.21.15.190:6819/12415","heartbeat_back_addr":"172.21.15.190:6823/12415","heartbeat_front_addr":"172.21.15.190:6821/12415","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:44.761435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:44.752911+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:44.784588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T12:37:53.322 INFO:tasks.ceph.ceph_manager.c2:all up! 2020-11-09T12:37:53.323 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-09T12:37:53.608 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:53.608 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":13,"fsid":"85b2f53f-98d9-4eb9-9d39-99874b789248","created":"2020-11-09T12:37:41.672623+0000","modified":"2020-11-09T12:37:52.753068+0000","last_up_change":"2020-11-09T12:37:46.696522+0000","last_in_change":"2020-11-09T12:37:46.696522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-09T12:37:45.570105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-09T12:37:50.150053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7a7f97f9-f92b-4d6e-8086-be39e96e43d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":12414},{"type":"v1","addr":"172.21.15.190:6801","nonce":12414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":12414},{"type":"v1","addr":"172.21.15.190:6803","nonce":12414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":12414},{"type":"v1","addr":"172.21.15.190:6807","nonce":12414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":12414},{"type":"v1","addr":"172.21.15.190:6805","nonce":12414}]},"public_addr":"172.21.15.190:6801/12414","cluster_addr":"172.21.15.190:6803/12414","heartbeat_back_addr":"172.21.15.190:6807/12414","heartbeat_front_addr":"172.21.15.190:6805/12414","state":["exists","up"]},{"osd":1,"uuid":"bc0af409-4279-46e6-af45-b0eab50c353e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":12416},{"type":"v1","addr":"172.21.15.190:6809","nonce":12416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":12416},{"type":"v1","addr":"172.21.15.190:6811","nonce":12416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":12416},{"type":"v1","addr":"172.21.15.190:6816","nonce":12416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":12416},{"type":"v1","addr":"172.21.15.190:6813","nonce":12416}]},"public_addr":"172.21.15.190:6809/12416","cluster_addr":"172.21.15.190:6811/12416","heartbeat_back_addr":"172.21.15.190:6816/12416","heartbeat_front_addr":"172.21.15.190:6813/12416","state":["exists","up"]},{"osd":2,"uuid":"03a55901-9fae-41f9-9b37-1cf54c976044","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":12415},{"type":"v1","addr":"172.21.15.190:6817","nonce":12415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":12415},{"type":"v1","addr":"172.21.15.190:6819","nonce":12415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":12415},{"type":"v1","addr":"172.21.15.190:6823","nonce":12415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":12415},{"type":"v1","addr":"172.21.15.190:6821","nonce":12415}]},"public_addr":"172.21.15.190:6817/12415","cluster_addr":"172.21.15.190:6819/12415","heartbeat_back_addr":"172.21.15.190:6823/12415","heartbeat_front_addr":"172.21.15.190:6821/12415","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:44.761435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:44.752911+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-09T12:37:44.784588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-09T12:37:53.622 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2020-11-09T12:37:53.792 INFO:teuthology.orchestra.run.smithi190.stdout:34359738372 2020-11-09T12:37:53.793 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2020-11-09T12:37:53.959 INFO:teuthology.orchestra.run.smithi190.stdout:34359738371 2020-11-09T12:37:53.960 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2020-11-09T12:37:54.125 INFO:teuthology.orchestra.run.smithi190.stdout:34359738372 2020-11-09T12:37:54.126 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2020-11-09T12:37:54.409 INFO:teuthology.orchestra.run.smithi190.stdout:34359738371 2020-11-09T12:37:54.423 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.0 2020-11-09T12:37:55.425 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2020-11-09T12:37:55.729 INFO:teuthology.orchestra.run.smithi190.stdout:34359738371 2020-11-09T12:37:55.743 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.0 2020-11-09T12:37:56.744 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2020-11-09T12:37:57.041 INFO:teuthology.orchestra.run.smithi190.stdout:34359738372 2020-11-09T12:37:57.054 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.0 2020-11-09T12:37:57.054 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2020-11-09T12:37:57.339 INFO:teuthology.orchestra.run.smithi190.stdout:34359738371 2020-11-09T12:37:57.353 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2020-11-09T12:37:57.353 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2020-11-09T12:37:57.639 INFO:teuthology.orchestra.run.smithi190.stdout:34359738372 2020-11-09T12:37:57.658 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2020-11-09T12:37:57.658 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-11-09T12:37:57.659 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2020-11-09T12:37:57.942 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:57.944 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2020-11-09T12:37:57.956 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-09T12:37:57.541185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14752,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558048,"statfs":{"total":322122547200,"available":322107441152,"internally_reserved":0,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.785785"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.770911+0000","last_change":"2020-11-09T12:37:51.769694+0000","last_active":"2020-11-09T12:37:51.770911+0000","last_peered":"2020-11-09T12:37:51.770911+0000","last_clean":"2020-11-09T12:37:51.770911+0000","last_became_active":"2020-11-09T12:37:51.769282+0000","last_became_peered":"2020-11-09T12:37:51.769282+0000","last_unstale":"2020-11-09T12:37:51.770911+0000","last_undegraded":"2020-11-09T12:37:51.770911+0000","last_fullsized":"2020-11-09T12:37:51.770911+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.770234+0000","last_change":"2020-11-09T12:37:51.769218+0000","last_active":"2020-11-09T12:37:51.770234+0000","last_peered":"2020-11-09T12:37:51.770234+0000","last_clean":"2020-11-09T12:37:51.770234+0000","last_became_active":"2020-11-09T12:37:51.768831+0000","last_became_peered":"2020-11-09T12:37:51.768831+0000","last_unstale":"2020-11-09T12:37:51.770234+0000","last_undegraded":"2020-11-09T12:37:51.770234+0000","last_fullsized":"2020-11-09T12:37:51.770234+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.767479+0000","last_change":"2020-11-09T12:37:51.766282+0000","last_active":"2020-11-09T12:37:51.767479+0000","last_peered":"2020-11-09T12:37:51.767479+0000","last_clean":"2020-11-09T12:37:51.767479+0000","last_became_active":"2020-11-09T12:37:51.765844+0000","last_became_peered":"2020-11-09T12:37:51.765844+0000","last_unstale":"2020-11-09T12:37:51.767479+0000","last_undegraded":"2020-11-09T12:37:51.767479+0000","last_fullsized":"2020-11-09T12:37:51.767479+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.769184+0000","last_change":"2020-11-09T12:37:51.767304+0000","last_active":"2020-11-09T12:37:51.769184+0000","last_peered":"2020-11-09T12:37:51.769184+0000","last_clean":"2020-11-09T12:37:51.769184+0000","last_became_active":"2020-11-09T12:37:51.766789+0000","last_became_peered":"2020-11-09T12:37:51.766789+0000","last_unstale":"2020-11-09T12:37:51.769184+0000","last_undegraded":"2020-11-09T12:37:51.769184+0000","last_fullsized":"2020-11-09T12:37:51.769184+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.764926+0000","last_change":"2020-11-09T12:37:47.812057+0000","last_active":"2020-11-09T12:37:51.764926+0000","last_peered":"2020-11-09T12:37:51.764926+0000","last_clean":"2020-11-09T12:37:51.764926+0000","last_became_active":"2020-11-09T12:37:47.811067+0000","last_became_peered":"2020-11-09T12:37:47.811067+0000","last_unstale":"2020-11-09T12:37:51.764926+0000","last_undegraded":"2020-11-09T12:37:51.764926+0000","last_fullsized":"2020-11-09T12:37:51.764926+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:45.688852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:45.688852+0000","last_clean_scrub_stamp":"2020-11-09T12:37:45.688852+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.769460+0000","last_change":"2020-11-09T12:37:51.767711+0000","last_active":"2020-11-09T12:37:51.769460+0000","last_peered":"2020-11-09T12:37:51.769460+0000","last_clean":"2020-11-09T12:37:51.769460+0000","last_became_active":"2020-11-09T12:37:51.767273+0000","last_became_peered":"2020-11-09T12:37:51.767273+0000","last_unstale":"2020-11-09T12:37:51.769460+0000","last_undegraded":"2020-11-09T12:37:51.769460+0000","last_fullsized":"2020-11-09T12:37:51.769460+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.768456+0000","last_change":"2020-11-09T12:37:51.766471+0000","last_active":"2020-11-09T12:37:51.768456+0000","last_peered":"2020-11-09T12:37:51.768456+0000","last_clean":"2020-11-09T12:37:51.768456+0000","last_became_active":"2020-11-09T12:37:51.766025+0000","last_became_peered":"2020-11-09T12:37:51.766025+0000","last_unstale":"2020-11-09T12:37:51.768456+0000","last_undegraded":"2020-11-09T12:37:51.768456+0000","last_fullsized":"2020-11-09T12:37:51.768456+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.770084+0000","last_change":"2020-11-09T12:37:51.768319+0000","last_active":"2020-11-09T12:37:51.770084+0000","last_peered":"2020-11-09T12:37:51.770084+0000","last_clean":"2020-11-09T12:37:51.770084+0000","last_became_active":"2020-11-09T12:37:51.767892+0000","last_became_peered":"2020-11-09T12:37:51.767892+0000","last_unstale":"2020-11-09T12:37:51.770084+0000","last_undegraded":"2020-11-09T12:37:51.770084+0000","last_fullsized":"2020-11-09T12:37:51.770084+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.767465+0000","last_change":"2020-11-09T12:37:51.766390+0000","last_active":"2020-11-09T12:37:51.767465+0000","last_peered":"2020-11-09T12:37:51.767465+0000","last_clean":"2020-11-09T12:37:51.767465+0000","last_became_active":"2020-11-09T12:37:51.765994+0000","last_became_peered":"2020-11-09T12:37:51.765994+0000","last_unstale":"2020-11-09T12:37:51.767465+0000","last_undegraded":"2020-11-09T12:37:51.767465+0000","last_fullsized":"2020-11-09T12:37:51.767465+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2010000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5149999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.423}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]}]}]}} 2020-11-09T12:37:57.957 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2020-11-09T12:37:58.239 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:58.242 INFO:teuthology.orchestra.run.smithi190.stderr:dumped all 2020-11-09T12:37:58.253 INFO:teuthology.orchestra.run.smithi190.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-09T12:37:57.541185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14752,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558048,"statfs":{"total":322122547200,"available":322107441152,"internally_reserved":0,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.785785"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.770911+0000","last_change":"2020-11-09T12:37:51.769694+0000","last_active":"2020-11-09T12:37:51.770911+0000","last_peered":"2020-11-09T12:37:51.770911+0000","last_clean":"2020-11-09T12:37:51.770911+0000","last_became_active":"2020-11-09T12:37:51.769282+0000","last_became_peered":"2020-11-09T12:37:51.769282+0000","last_unstale":"2020-11-09T12:37:51.770911+0000","last_undegraded":"2020-11-09T12:37:51.770911+0000","last_fullsized":"2020-11-09T12:37:51.770911+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.770234+0000","last_change":"2020-11-09T12:37:51.769218+0000","last_active":"2020-11-09T12:37:51.770234+0000","last_peered":"2020-11-09T12:37:51.770234+0000","last_clean":"2020-11-09T12:37:51.770234+0000","last_became_active":"2020-11-09T12:37:51.768831+0000","last_became_peered":"2020-11-09T12:37:51.768831+0000","last_unstale":"2020-11-09T12:37:51.770234+0000","last_undegraded":"2020-11-09T12:37:51.770234+0000","last_fullsized":"2020-11-09T12:37:51.770234+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.767479+0000","last_change":"2020-11-09T12:37:51.766282+0000","last_active":"2020-11-09T12:37:51.767479+0000","last_peered":"2020-11-09T12:37:51.767479+0000","last_clean":"2020-11-09T12:37:51.767479+0000","last_became_active":"2020-11-09T12:37:51.765844+0000","last_became_peered":"2020-11-09T12:37:51.765844+0000","last_unstale":"2020-11-09T12:37:51.767479+0000","last_undegraded":"2020-11-09T12:37:51.767479+0000","last_fullsized":"2020-11-09T12:37:51.767479+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.769184+0000","last_change":"2020-11-09T12:37:51.767304+0000","last_active":"2020-11-09T12:37:51.769184+0000","last_peered":"2020-11-09T12:37:51.769184+0000","last_clean":"2020-11-09T12:37:51.769184+0000","last_became_active":"2020-11-09T12:37:51.766789+0000","last_became_peered":"2020-11-09T12:37:51.766789+0000","last_unstale":"2020-11-09T12:37:51.769184+0000","last_undegraded":"2020-11-09T12:37:51.769184+0000","last_fullsized":"2020-11-09T12:37:51.769184+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.764926+0000","last_change":"2020-11-09T12:37:47.812057+0000","last_active":"2020-11-09T12:37:51.764926+0000","last_peered":"2020-11-09T12:37:51.764926+0000","last_clean":"2020-11-09T12:37:51.764926+0000","last_became_active":"2020-11-09T12:37:47.811067+0000","last_became_peered":"2020-11-09T12:37:47.811067+0000","last_unstale":"2020-11-09T12:37:51.764926+0000","last_undegraded":"2020-11-09T12:37:51.764926+0000","last_fullsized":"2020-11-09T12:37:51.764926+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:45.688852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:45.688852+0000","last_clean_scrub_stamp":"2020-11-09T12:37:45.688852+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.769460+0000","last_change":"2020-11-09T12:37:51.767711+0000","last_active":"2020-11-09T12:37:51.769460+0000","last_peered":"2020-11-09T12:37:51.769460+0000","last_clean":"2020-11-09T12:37:51.769460+0000","last_became_active":"2020-11-09T12:37:51.767273+0000","last_became_peered":"2020-11-09T12:37:51.767273+0000","last_unstale":"2020-11-09T12:37:51.769460+0000","last_undegraded":"2020-11-09T12:37:51.769460+0000","last_fullsized":"2020-11-09T12:37:51.769460+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.768456+0000","last_change":"2020-11-09T12:37:51.766471+0000","last_active":"2020-11-09T12:37:51.768456+0000","last_peered":"2020-11-09T12:37:51.768456+0000","last_clean":"2020-11-09T12:37:51.768456+0000","last_became_active":"2020-11-09T12:37:51.766025+0000","last_became_peered":"2020-11-09T12:37:51.766025+0000","last_unstale":"2020-11-09T12:37:51.768456+0000","last_undegraded":"2020-11-09T12:37:51.768456+0000","last_fullsized":"2020-11-09T12:37:51.768456+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.770084+0000","last_change":"2020-11-09T12:37:51.768319+0000","last_active":"2020-11-09T12:37:51.770084+0000","last_peered":"2020-11-09T12:37:51.770084+0000","last_clean":"2020-11-09T12:37:51.770084+0000","last_became_active":"2020-11-09T12:37:51.767892+0000","last_became_peered":"2020-11-09T12:37:51.767892+0000","last_unstale":"2020-11-09T12:37:51.770084+0000","last_undegraded":"2020-11-09T12:37:51.770084+0000","last_fullsized":"2020-11-09T12:37:51.770084+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-09T12:37:51.767465+0000","last_change":"2020-11-09T12:37:51.766390+0000","last_active":"2020-11-09T12:37:51.767465+0000","last_peered":"2020-11-09T12:37:51.767465+0000","last_clean":"2020-11-09T12:37:51.767465+0000","last_became_active":"2020-11-09T12:37:51.765994+0000","last_became_peered":"2020-11-09T12:37:51.765994+0000","last_unstale":"2020-11-09T12:37:51.767465+0000","last_undegraded":"2020-11-09T12:37:51.767465+0000","last_fullsized":"2020-11-09T12:37:51.767465+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-09T12:37:50.734695+0000","last_clean_scrub_stamp":"2020-11-09T12:37:50.734695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2010000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5149999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.423}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]}]}]}} 2020-11-09T12:37:58.254 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-11-09T12:37:58.254 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-11-09T12:37:58.255 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2020-11-09T12:37:58.255 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2020-11-09T12:37:58.588 INFO:teuthology.orchestra.run.smithi190.stdout: 2020-11-09T12:37:58.602 INFO:teuthology.orchestra.run.smithi190.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-09T12:37:58.602 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2020-11-09T12:37:58.603 INFO:teuthology.run_tasks:Running task rgw... 2020-11-09T12:37:58.616 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2020-11-09T12:37:58.616 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2020-11-09T12:37:58.617 INFO:tasks.rgw:Creating data pools 2020-11-09T12:37:58.617 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-11-09T12:37:58.617 INFO:teuthology.orchestra.run.smithi164:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-11-09T12:37:59.009 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'default.rgw.buckets.data' created 2020-11-09T12:37:59.024 INFO:teuthology.orchestra.run.smithi164:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-11-09T12:38:00.028 INFO:teuthology.orchestra.run.smithi164.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-09T12:38:00.051 INFO:teuthology.orchestra.run.smithi164:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-11-09T12:38:01.026 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'default.rgw.buckets.index' created 2020-11-09T12:38:01.043 INFO:teuthology.orchestra.run.smithi164:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-11-09T12:38:02.032 INFO:teuthology.orchestra.run.smithi164.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-09T12:38:02.058 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-11-09T12:38:02.059 INFO:teuthology.orchestra.run.smithi164:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-11-09T12:38:02.337 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'default.rgw.buckets.data' already exists 2020-11-09T12:38:02.355 INFO:teuthology.orchestra.run.smithi164:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-11-09T12:38:03.022 INFO:teuthology.orchestra.run.smithi164.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-09T12:38:03.036 INFO:teuthology.orchestra.run.smithi164:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-11-09T12:38:03.337 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'default.rgw.buckets.index' already exists 2020-11-09T12:38:03.347 INFO:teuthology.orchestra.run.smithi164:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-11-09T12:38:04.026 INFO:teuthology.orchestra.run.smithi164.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-09T12:38:04.039 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-11-09T12:38:04.040 INFO:teuthology.orchestra.run.smithi190:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-11-09T12:38:04.687 INFO:teuthology.orchestra.run.smithi190.stderr:pool 'default.rgw.buckets.data' created 2020-11-09T12:38:04.701 INFO:teuthology.orchestra.run.smithi190:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-11-09T12:38:05.704 INFO:teuthology.orchestra.run.smithi190.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-09T12:38:05.722 INFO:teuthology.orchestra.run.smithi190:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-11-09T12:38:06.711 INFO:teuthology.orchestra.run.smithi190.stderr:pool 'default.rgw.buckets.index' created 2020-11-09T12:38:06.717 INFO:teuthology.orchestra.run.smithi190:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-11-09T12:38:07.722 INFO:teuthology.orchestra.run.smithi190.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-09T12:38:07.749 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-11-09T12:38:07.750 INFO:teuthology.orchestra.run.smithi190:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-11-09T12:38:08.033 INFO:teuthology.orchestra.run.smithi190.stderr:pool 'default.rgw.buckets.data' already exists 2020-11-09T12:38:08.043 INFO:teuthology.orchestra.run.smithi190:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-11-09T12:38:08.697 INFO:teuthology.orchestra.run.smithi190.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-09T12:38:08.710 INFO:teuthology.orchestra.run.smithi190:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-11-09T12:38:09.006 INFO:teuthology.orchestra.run.smithi190.stderr:pool 'default.rgw.buckets.index' already exists 2020-11-09T12:38:09.023 INFO:teuthology.orchestra.run.smithi190:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-11-09T12:38:09.718 INFO:teuthology.orchestra.run.smithi190.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-09T12:38:09.731 DEBUG:tasks.rgw:Pools created 2020-11-09T12:38:09.732 INFO:tasks.rgw:Configuring compression type = random 2020-11-09T12:38:09.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-11-09T12:38:09.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-11-09T12:38:09.733 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-11-09T12:38:09.770 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:09.771 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:09.791 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:09.783+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:10.995+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:10.995+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:10.999+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:10.999+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:10.999+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:10.999+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:10.999+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:11.018 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.019+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.019+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.019+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:11.020 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.019+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:11.020 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.019+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:11.020 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.019+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.019+0000 7f1d2dcfe0c0 10 cannot find current period zonegroup using local zonegroup 2020-11-09T12:38:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.019+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.019+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.019+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.023+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.023+0000 7f1d2dcfe0c0 10 Creating default zonegroup 2020-11-09T12:38:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.023+0000 7f1d2dcfe0c0 20 get_system_obj_state: rctx=0x7ffdbeb05e50 obj=:.pools.avail state=0x560311b4e2e0 s->prefetch_data=0 2020-11-09T12:38:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.023+0000 7f1d2dcfe0c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-11-09T12:38:11.037 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.039+0000 7f1d2dcfe0c0 10 WARNING: store->list_zones() returned r=-2 2020-11-09T12:38:11.037 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.039+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:11.038 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.039+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:11.042 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.043+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:11.043 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.043+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:11.043 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.043+0000 7f1d2dcfe0c0 10 could not read realm id: (2) No such file or directory 2020-11-09T12:38:11.043 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.043+0000 7f1d2dcfe0c0 10 WARNING: failed to set zone as default, r=-22 2020-11-09T12:38:11.043 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.043+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:11.044 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.043+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:11.049 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.051+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:11.050 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.051+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-09T12:38:11.050 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.051+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:11.051 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.051+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T12:38:11.051 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.051+0000 7f1d2dcfe0c0 20 zonegroup default 2020-11-09T12:38:11.051 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.051+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:11.051 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.051+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:11.052 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.051+0000 7f1d2dcfe0c0 10 Cannot find current period zone using local zone 2020-11-09T12:38:11.052 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.051+0000 7f1d2dcfe0c0 10 Using default name default 2020-11-09T12:38:11.052 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.051+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:11.052 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.055+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:11.053 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.055+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:11.053 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.055+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-09T12:38:11.053 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.055+0000 7f1d2dcfe0c0 20 zone default found 2020-11-09T12:38:11.053 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:11.055+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:13.034 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:13.035+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:13.034 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:13.035+0000 7f1d2dcfe0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:13.035 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:13.035+0000 7f1d2dcfe0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:13.035 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:13.035+0000 7f1d2dcfe0c0 20 started sync module instance, tier type = 2020-11-09T12:38:13.035 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:13.035+0000 7f1d2dcfe0c0 20 started zone id=35c3bdcf-6e82-4fdb-b9c6-4b1f35f33540 (name=default) with tier type = 2020-11-09T12:38:15.070 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.071+0000 7f1d2dcfe0c0 20 add_watcher() i=0 2020-11-09T12:38:15.070 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.071+0000 7f1d2dcfe0c0 20 add_watcher() i=1 2020-11-09T12:38:15.070 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.071+0000 7f1d2dcfe0c0 20 add_watcher() i=2 2020-11-09T12:38:15.070 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.071+0000 7f1d2dcfe0c0 20 add_watcher() i=3 2020-11-09T12:38:15.071 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.071+0000 7f1d2dcfe0c0 20 add_watcher() i=4 2020-11-09T12:38:15.071 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.071+0000 7f1d2dcfe0c0 20 add_watcher() i=5 2020-11-09T12:38:15.073 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.075+0000 7f1d2dcfe0c0 20 add_watcher() i=6 2020-11-09T12:38:15.073 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.075+0000 7f1d2dcfe0c0 20 add_watcher() i=7 2020-11-09T12:38:15.073 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.075+0000 7f1d2dcfe0c0 2 all 8 watchers are set, enabling cache 2020-11-09T12:38:15.075 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.075+0000 7f1d2dcfe0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:15.075 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.075+0000 7f1d2dcfe0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:15.076 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.079+0000 7f1d2dcfe0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:15.076 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.079+0000 7f1d2dcfe0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:15.077 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.079+0000 7f1d2dcfe0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:15.079 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.079+0000 7f1d2dcfe0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:15.083 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.083+0000 7f1d2dcfe0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:15.085 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.087+0000 7f1d2dcfe0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:15.089 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.091+0000 7f1d2dcfe0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:15.092 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.095+0000 7f1d2dcfe0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:15.096 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.099+0000 7f1d2dcfe0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:15.099 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.099+0000 7f1d2dcfe0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:15.103 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.103+0000 7f1ceb7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T12:38:15.103 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.103+0000 7f1d2dcfe0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T12:38:15.104 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:15.103+0000 7f1d2dcfe0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T12:38:19.045 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.047+0000 7f1d2dcfe0c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-11-09T12:38:19.262 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.263+0000 7f1c567fc700 20 reqs_thread_entry: start 2020-11-09T12:38:19.263 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.263+0000 7f1d2dcfe0c0 20 init_complete bucket index max shards: 11 2020-11-09T12:38:19.263 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.263+0000 7f1c3ffff700 20 reqs_thread_entry: start 2020-11-09T12:38:19.263 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.263+0000 7f1d2dcfe0c0 10 Started notification manager with: 1 workers 2020-11-09T12:38:19.263 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.263+0000 7f1c3e7fc700 20 INFO: next queues processing will happen at: Mon Nov 9 12:38:49 2020 2020-11-09T12:38:19.263 INFO:teuthology.orchestra.run.smithi164.stderr: 2020-11-09T12:38:19.263 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.263+0000 7f1c3dffb700 20 reqs_thread_entry: start 2020-11-09T12:38:19.294 INFO:teuthology.orchestra.run.smithi164.stdout:[] 2020-11-09T12:38:19.294 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.295+0000 7f1d2dcfe0c0 20 remove_watcher() i=0 2020-11-09T12:38:19.294 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.295+0000 7f1d2dcfe0c0 2 removed watcher, disabling cache 2020-11-09T12:38:19.296 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.299+0000 7f1d2dcfe0c0 20 remove_watcher() i=1 2020-11-09T12:38:19.299 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.299+0000 7f1d2dcfe0c0 20 remove_watcher() i=2 2020-11-09T12:38:19.302 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.303+0000 7f1d2dcfe0c0 20 remove_watcher() i=3 2020-11-09T12:38:19.305 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.307+0000 7f1d2dcfe0c0 20 remove_watcher() i=4 2020-11-09T12:38:19.307 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.307+0000 7f1d2dcfe0c0 20 remove_watcher() i=5 2020-11-09T12:38:19.310 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.311+0000 7f1d2dcfe0c0 20 remove_watcher() i=6 2020-11-09T12:38:19.313 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.315+0000 7f1d2dcfe0c0 20 remove_watcher() i=7 2020-11-09T12:38:19.314 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.315+0000 7f1ceb7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T12:38:19.357 DEBUG:tasks.util.rgw: json result: [] 2020-11-09T12:38:19.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T12:38:19.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T12:38:19.358 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-09T12:38:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:19.456 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.459+0000 7fac192fa0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T12:38:19.456 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.459+0000 7fac192fa0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T12:38:19.462 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.463+0000 7fab4e7fc700 20 reqs_thread_entry: start 2020-11-09T12:38:19.463 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.463+0000 7fac192fa0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.463 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.463+0000 7fac192fa0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:19.463 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.463+0000 7fac192fa0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.465 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.467+0000 7fac192fa0c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-09T12:38:19.465 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.467+0000 7fac192fa0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.466 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.467+0000 7fac192fa0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:19.467 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.467+0000 7fac192fa0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.467 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.467+0000 7fac192fa0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:19.467 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.467+0000 7fac192fa0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.468 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.471+0000 7fac192fa0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T12:38:19.475 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:38:19.475 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "35c3bdcf-6e82-4fdb-b9c6-4b1f35f33540", 2020-11-09T12:38:19.476 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "default", 2020-11-09T12:38:19.476 INFO:teuthology.orchestra.run.smithi164.stdout: "domain_root": "default.rgw.meta:root", 2020-11-09T12:38:19.476 INFO:teuthology.orchestra.run.smithi164.stdout: "control_pool": "default.rgw.control", 2020-11-09T12:38:19.476 INFO:teuthology.orchestra.run.smithi164.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-09T12:38:19.476 INFO:teuthology.orchestra.run.smithi164.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-09T12:38:19.476 INFO:teuthology.orchestra.run.smithi164.stdout: "log_pool": "default.rgw.log", 2020-11-09T12:38:19.477 INFO:teuthology.orchestra.run.smithi164.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-09T12:38:19.477 INFO:teuthology.orchestra.run.smithi164.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-09T12:38:19.477 INFO:teuthology.orchestra.run.smithi164.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-09T12:38:19.477 INFO:teuthology.orchestra.run.smithi164.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-09T12:38:19.478 INFO:teuthology.orchestra.run.smithi164.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-09T12:38:19.478 INFO:teuthology.orchestra.run.smithi164.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-09T12:38:19.478 INFO:teuthology.orchestra.run.smithi164.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-09T12:38:19.478 INFO:teuthology.orchestra.run.smithi164.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-09T12:38:19.478 INFO:teuthology.orchestra.run.smithi164.stdout: "otp_pool": "default.rgw.otp", 2020-11-09T12:38:19.479 INFO:teuthology.orchestra.run.smithi164.stdout: "system_key": { 2020-11-09T12:38:19.479 INFO:teuthology.orchestra.run.smithi164.stdout: "access_key": "", 2020-11-09T12:38:19.479 INFO:teuthology.orchestra.run.smithi164.stdout: "secret_key": "" 2020-11-09T12:38:19.479 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:38:19.479 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_pools": [ 2020-11-09T12:38:19.479 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:38:19.480 INFO:teuthology.orchestra.run.smithi164.stdout: "key": "default-placement", 2020-11-09T12:38:19.480 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:38:19.480 INFO:teuthology.orchestra.run.smithi164.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-09T12:38:19.480 INFO:teuthology.orchestra.run.smithi164.stdout: "storage_classes": { 2020-11-09T12:38:19.480 INFO:teuthology.orchestra.run.smithi164.stdout: "STANDARD": { 2020-11-09T12:38:19.480 INFO:teuthology.orchestra.run.smithi164.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-09T12:38:19.480 INFO:teuthology.orchestra.run.smithi164.stdout: "compression_type": "random" 2020-11-09T12:38:19.481 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:19.481 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:38:19.481 INFO:teuthology.orchestra.run.smithi164.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-09T12:38:19.481 INFO:teuthology.orchestra.run.smithi164.stdout: "index_type": 0 2020-11-09T12:38:19.481 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:19.481 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:19.481 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:38:19.481 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "", 2020-11-09T12:38:19.482 INFO:teuthology.orchestra.run.smithi164.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-09T12:38:19.482 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:38:19.483 DEBUG:tasks.util.rgw: json result: {'id': '35c3bdcf-6e82-4fdb-b9c6-4b1f35f33540', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-09T12:38:19.483 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-11-09T12:38:19.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-11-09T12:38:19.484 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-11-09T12:38:19.557 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:19.557 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:19.568 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.571+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.570 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.571+0000 7f6ecef2e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:19.570 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.571+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.571 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.571+0000 7f6ecef2e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:19.571 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.571+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.571 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.571+0000 7f6ecef2e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:19.571 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.571+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.572 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.575+0000 7f6ecef2e0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:19.598 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.599+0000 7f6ecef2e0c0 20 RGWRados::pool_iterate: got zonegroup_info.094b03ea-eeb2-4fb3-92e2-c74da76bd620 2020-11-09T12:38:19.598 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.599+0000 7f6ecef2e0c0 20 RGWRados::pool_iterate: got zone_info.35c3bdcf-6e82-4fdb-b9c6-4b1f35f33540 2020-11-09T12:38:19.598 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.599+0000 7f6ecef2e0c0 20 RGWRados::pool_iterate: got zone_names.default 2020-11-09T12:38:19.599 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.599+0000 7f6ecef2e0c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-11-09T12:38:19.599 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.599+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.599 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.599+0000 7f6ecef2e0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:19.599 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.599+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.600 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.603+0000 7f6ecef2e0c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-09T12:38:19.600 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.603+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.603+0000 7f6ecef2e0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.603+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.603+0000 7f6ecef2e0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T12:38:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.603+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.603+0000 7f6ecef2e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:19.603 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.603+0000 7f6ecef2e0c0 10 cannot find current period zonegroup using local zonegroup 2020-11-09T12:38:19.603 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.603+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.604 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.603+0000 7f6ecef2e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:19.604 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.603+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.604 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.607+0000 7f6ecef2e0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:19.604 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.607+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.605 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.607+0000 7f6ecef2e0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T12:38:19.605 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.607+0000 7f6ecef2e0c0 20 zonegroup default 2020-11-09T12:38:19.605 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.607+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.606 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.607+0000 7f6ecef2e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:19.606 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.607+0000 7f6ecef2e0c0 10 Cannot find current period zone using local zone 2020-11-09T12:38:19.606 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.607+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.607 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.607+0000 7f6ecef2e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:19.607 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.607+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.608 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.607+0000 7f6ecef2e0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:19.608 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.607+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.610 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.611+0000 7f6ecef2e0c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-09T12:38:19.610 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.611+0000 7f6ecef2e0c0 20 zone default found 2020-11-09T12:38:19.611 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.611+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.611 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.611+0000 7f6ecef2e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:19.611 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.611+0000 7f6ecef2e0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.612 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.611+0000 7f6ecef2e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:19.612 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.611+0000 7f6ecef2e0c0 20 started sync module instance, tier type = 2020-11-09T12:38:19.612 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.611+0000 7f6ecef2e0c0 20 started zone id=35c3bdcf-6e82-4fdb-b9c6-4b1f35f33540 (name=default) with tier type = 2020-11-09T12:38:19.641 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.643+0000 7f6ecef2e0c0 20 add_watcher() i=0 2020-11-09T12:38:19.641 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.643+0000 7f6ecef2e0c0 20 add_watcher() i=1 2020-11-09T12:38:19.642 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.643+0000 7f6ecef2e0c0 20 add_watcher() i=2 2020-11-09T12:38:19.642 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.643+0000 7f6ecef2e0c0 20 add_watcher() i=3 2020-11-09T12:38:19.642 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.643+0000 7f6ecef2e0c0 20 add_watcher() i=4 2020-11-09T12:38:19.642 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.643+0000 7f6ecef2e0c0 20 add_watcher() i=5 2020-11-09T12:38:19.643 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.643+0000 7f6ecef2e0c0 20 add_watcher() i=6 2020-11-09T12:38:19.643 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.643+0000 7f6ecef2e0c0 20 add_watcher() i=7 2020-11-09T12:38:19.644 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.643+0000 7f6ecef2e0c0 2 all 8 watchers are set, enabling cache 2020-11-09T12:38:19.645 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.647+0000 7f6ecef2e0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:19.646 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.647+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T12:38:19.646 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.647+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T12:38:19.647 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.647+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T12:38:19.647 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.647+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T12:38:19.647 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.647+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T12:38:19.647 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.647+0000 7f6ecef2e0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:19.648 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.647+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T12:38:19.648 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.647+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T12:38:19.648 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.651+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T12:38:19.649 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.651+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T12:38:19.649 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.651+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T12:38:19.649 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.651+0000 7f6ecef2e0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:19.649 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.651+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T12:38:19.650 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.651+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T12:38:19.650 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.651+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T12:38:19.650 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.651+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T12:38:19.651 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.651+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T12:38:19.651 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.651+0000 7f6ecef2e0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:19.651 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.651+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T12:38:19.652 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.651+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T12:38:19.652 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.651+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T12:38:19.652 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.655+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T12:38:19.652 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.655+0000 7f6ecef2e0c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T12:38:19.653 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.655+0000 7f6ecef2e0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:19.654 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.655+0000 7f6ecef2e0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:19.655 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.655+0000 7f6ecef2e0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:19.657 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.659+0000 7f6ecef2e0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:19.658 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.659+0000 7f6ecef2e0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:19.660 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.663+0000 7f6ecef2e0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:19.660 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.663+0000 7f6ecef2e0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:19.663 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.663+0000 7f6ecef2e0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:19.663 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.663+0000 7f6e8bfe7700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T12:38:19.664 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.667+0000 7f6ecef2e0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T12:38:19.664 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.667+0000 7f6ecef2e0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T12:38:19.755 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.755+0000 7f6dfaffd700 20 reqs_thread_entry: start 2020-11-09T12:38:19.755 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.755+0000 7f6df8ff9700 20 reqs_thread_entry: start 2020-11-09T12:38:19.755 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.755+0000 7f6ecef2e0c0 20 init_complete bucket index max shards: 11 2020-11-09T12:38:19.756 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.755+0000 7f6ecef2e0c0 10 Started notification manager with: 1 workers 2020-11-09T12:38:19.756 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.755+0000 7f6deeffd700 20 INFO: next queues processing will happen at: Mon Nov 9 12:38:50 2020 2020-11-09T12:38:19.756 INFO:teuthology.orchestra.run.smithi164.stderr: 2020-11-09T12:38:19.756 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.755+0000 7f6dee7fc700 20 reqs_thread_entry: start 2020-11-09T12:38:19.785 INFO:teuthology.orchestra.run.smithi164.stdout:[] 2020-11-09T12:38:19.786 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.787+0000 7f6ecef2e0c0 20 remove_watcher() i=0 2020-11-09T12:38:19.786 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.787+0000 7f6ecef2e0c0 2 removed watcher, disabling cache 2020-11-09T12:38:19.789 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.791+0000 7f6ecef2e0c0 20 remove_watcher() i=1 2020-11-09T12:38:19.792 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.791+0000 7f6ecef2e0c0 20 remove_watcher() i=2 2020-11-09T12:38:19.795 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.795+0000 7f6ecef2e0c0 20 remove_watcher() i=3 2020-11-09T12:38:19.797 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.799+0000 7f6ecef2e0c0 20 remove_watcher() i=4 2020-11-09T12:38:19.800 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.803+0000 7f6ecef2e0c0 20 remove_watcher() i=5 2020-11-09T12:38:19.802 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.803+0000 7f6ecef2e0c0 20 remove_watcher() i=6 2020-11-09T12:38:19.805 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.807+0000 7f6ecef2e0c0 20 remove_watcher() i=7 2020-11-09T12:38:19.806 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.807+0000 7f6e8bfe7700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T12:38:19.812 DEBUG:tasks.util.rgw: json result: [] 2020-11-09T12:38:19.813 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T12:38:19.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T12:38:19.813 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-09T12:38:19.896 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:19.896 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:19.906 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.907+0000 7f8df72520c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T12:38:19.906 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.907+0000 7f8df72520c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T12:38:19.912 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.915+0000 7f8df72520c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.913 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.915+0000 7f8d2bfff700 20 reqs_thread_entry: start 2020-11-09T12:38:19.914 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.915+0000 7f8df72520c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:19.914 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.915+0000 7f8df72520c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.916 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.919+0000 7f8df72520c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-09T12:38:19.916 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.919+0000 7f8df72520c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.916 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.919+0000 7f8df72520c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:19.917 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.919+0000 7f8df72520c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.917 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.919+0000 7f8df72520c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:19.917 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.919+0000 7f8df72520c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:19.918 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:19.919+0000 7f8df72520c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T12:38:19.926 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:38:19.926 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "35c3bdcf-6e82-4fdb-b9c6-4b1f35f33540", 2020-11-09T12:38:19.927 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "default", 2020-11-09T12:38:19.927 INFO:teuthology.orchestra.run.smithi164.stdout: "domain_root": "default.rgw.meta:root", 2020-11-09T12:38:19.927 INFO:teuthology.orchestra.run.smithi164.stdout: "control_pool": "default.rgw.control", 2020-11-09T12:38:19.927 INFO:teuthology.orchestra.run.smithi164.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-09T12:38:19.927 INFO:teuthology.orchestra.run.smithi164.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-09T12:38:19.927 INFO:teuthology.orchestra.run.smithi164.stdout: "log_pool": "default.rgw.log", 2020-11-09T12:38:19.927 INFO:teuthology.orchestra.run.smithi164.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-09T12:38:19.928 INFO:teuthology.orchestra.run.smithi164.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-09T12:38:19.928 INFO:teuthology.orchestra.run.smithi164.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-09T12:38:19.928 INFO:teuthology.orchestra.run.smithi164.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-09T12:38:19.928 INFO:teuthology.orchestra.run.smithi164.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-09T12:38:19.928 INFO:teuthology.orchestra.run.smithi164.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-09T12:38:19.928 INFO:teuthology.orchestra.run.smithi164.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-09T12:38:19.928 INFO:teuthology.orchestra.run.smithi164.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-09T12:38:19.929 INFO:teuthology.orchestra.run.smithi164.stdout: "otp_pool": "default.rgw.otp", 2020-11-09T12:38:19.929 INFO:teuthology.orchestra.run.smithi164.stdout: "system_key": { 2020-11-09T12:38:19.929 INFO:teuthology.orchestra.run.smithi164.stdout: "access_key": "", 2020-11-09T12:38:19.929 INFO:teuthology.orchestra.run.smithi164.stdout: "secret_key": "" 2020-11-09T12:38:19.929 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:38:19.929 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_pools": [ 2020-11-09T12:38:19.929 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:38:19.929 INFO:teuthology.orchestra.run.smithi164.stdout: "key": "default-placement", 2020-11-09T12:38:19.930 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:38:19.930 INFO:teuthology.orchestra.run.smithi164.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-09T12:38:19.930 INFO:teuthology.orchestra.run.smithi164.stdout: "storage_classes": { 2020-11-09T12:38:19.930 INFO:teuthology.orchestra.run.smithi164.stdout: "STANDARD": { 2020-11-09T12:38:19.930 INFO:teuthology.orchestra.run.smithi164.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-09T12:38:19.930 INFO:teuthology.orchestra.run.smithi164.stdout: "compression_type": "random" 2020-11-09T12:38:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:38:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-09T12:38:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: "index_type": 0 2020-11-09T12:38:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:38:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "", 2020-11-09T12:38:19.932 INFO:teuthology.orchestra.run.smithi164.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-09T12:38:19.932 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:38:19.932 DEBUG:tasks.util.rgw: json result: {'id': '35c3bdcf-6e82-4fdb-b9c6-4b1f35f33540', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-09T12:38:19.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-11-09T12:38:19.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-11-09T12:38:19.933 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-11-09T12:38:19.968 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:19.969 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:19.982 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:19.984+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.757 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.760+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:21.761 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.760+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.762 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.760+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:21.762 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.760+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.762 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.760+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:21.763 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.760+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.763 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.760+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:21.783 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.784+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.783 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.784+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:21.784 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.784+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.784 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.784+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:21.784 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.784+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.784 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.784+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:21.785 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.784+0000 7fc9f585a0c0 10 cannot find current period zonegroup using local zonegroup 2020-11-09T12:38:21.785 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.784+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.785 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.784+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:21.785 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.784+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.786 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.788+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:21.787 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.788+0000 7fc9f585a0c0 10 Creating default zonegroup 2020-11-09T12:38:21.787 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.788+0000 7fc9f585a0c0 20 get_system_obj_state: rctx=0x7ffc83cf72b0 obj=:.pools.avail state=0x55b1a3edd2e0 s->prefetch_data=0 2020-11-09T12:38:21.787 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.788+0000 7fc9f585a0c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-11-09T12:38:21.800 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.800+0000 7fc9f585a0c0 10 WARNING: store->list_zones() returned r=-2 2020-11-09T12:38:21.800 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.800+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.801 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.800+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:21.803 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.808+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.804 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.808+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:21.804 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.808+0000 7fc9f585a0c0 10 could not read realm id: (2) No such file or directory 2020-11-09T12:38:21.804 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.808+0000 7fc9f585a0c0 10 WARNING: failed to set zone as default, r=-22 2020-11-09T12:38:21.804 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.808+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.805 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.808+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:21.809 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.812+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.810 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.812+0000 7fc9f585a0c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-09T12:38:21.812 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.812+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.812 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.812+0000 7fc9f585a0c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T12:38:21.812 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.812+0000 7fc9f585a0c0 20 zonegroup default 2020-11-09T12:38:21.812 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.812+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.813 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.816+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:21.813 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.816+0000 7fc9f585a0c0 10 Cannot find current period zone using local zone 2020-11-09T12:38:21.813 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.816+0000 7fc9f585a0c0 10 Using default name default 2020-11-09T12:38:21.814 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.816+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.814 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.816+0000 7fc9f585a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:21.814 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.816+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:21.814 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.816+0000 7fc9f585a0c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-09T12:38:21.814 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.816+0000 7fc9f585a0c0 20 zone default found 2020-11-09T12:38:21.814 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:21.816+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:23.774 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:23.776+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:23.774 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:23.776+0000 7fc9f585a0c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:23.775 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:23.776+0000 7fc9f585a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:23.775 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:23.776+0000 7fc9f585a0c0 20 started sync module instance, tier type = 2020-11-09T12:38:23.775 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:23.776+0000 7fc9f585a0c0 20 started zone id=e6c9450e-d772-4559-a326-d447647d396f (name=default) with tier type = 2020-11-09T12:38:25.828 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.832+0000 7fc9f585a0c0 20 add_watcher() i=0 2020-11-09T12:38:25.829 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.832+0000 7fc9f585a0c0 20 add_watcher() i=1 2020-11-09T12:38:25.829 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.832+0000 7fc9f585a0c0 20 add_watcher() i=2 2020-11-09T12:38:25.829 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.832+0000 7fc9f585a0c0 20 add_watcher() i=3 2020-11-09T12:38:25.829 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.832+0000 7fc9f585a0c0 20 add_watcher() i=4 2020-11-09T12:38:25.829 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.832+0000 7fc9f585a0c0 20 add_watcher() i=5 2020-11-09T12:38:25.829 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.832+0000 7fc9f585a0c0 20 add_watcher() i=6 2020-11-09T12:38:25.830 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.832+0000 7fc9f585a0c0 20 add_watcher() i=7 2020-11-09T12:38:25.830 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.832+0000 7fc9f585a0c0 2 all 8 watchers are set, enabling cache 2020-11-09T12:38:25.832 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.836+0000 7fc9f585a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:25.833 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.836+0000 7fc9f585a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:25.833 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.836+0000 7fc9f585a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:25.834 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.836+0000 7fc9f585a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:25.834 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.836+0000 7fc9f585a0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:25.837 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.840+0000 7fc9f585a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:25.841 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.844+0000 7fc9f585a0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:25.843 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.848+0000 7fc9f585a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:25.847 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.852+0000 7fc9f585a0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:25.850 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.852+0000 7fc9f585a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:25.854 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.856+0000 7fc9f585a0c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:25.857 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.860+0000 7fc9f585a0c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:25.859 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.860+0000 7fc9b27e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T12:38:25.859 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.864+0000 7fc9f585a0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T12:38:25.859 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:25.864+0000 7fc9f585a0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T12:38:29.787 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.786+0000 7fc9f585a0c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-11-09T12:38:29.909 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.910+0000 7fc9217fa700 20 reqs_thread_entry: start 2020-11-09T12:38:29.910 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.910+0000 7fc9f585a0c0 20 init_complete bucket index max shards: 11 2020-11-09T12:38:29.910 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.910+0000 7fc90effd700 20 reqs_thread_entry: start 2020-11-09T12:38:29.910 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.910+0000 7fc9f585a0c0 10 Started notification manager with: 1 workers 2020-11-09T12:38:29.910 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.910+0000 7fc90d7fa700 20 INFO: next queues processing will happen at: Mon Nov 9 12:39:00 2020 2020-11-09T12:38:29.910 INFO:teuthology.orchestra.run.smithi190.stderr: 2020-11-09T12:38:29.910 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.910+0000 7fc90cff9700 20 reqs_thread_entry: start 2020-11-09T12:38:29.938 INFO:teuthology.orchestra.run.smithi190.stdout:[] 2020-11-09T12:38:29.939 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.938+0000 7fc9f585a0c0 20 remove_watcher() i=0 2020-11-09T12:38:29.939 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.938+0000 7fc9f585a0c0 2 removed watcher, disabling cache 2020-11-09T12:38:29.941 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.942+0000 7fc9f585a0c0 20 remove_watcher() i=1 2020-11-09T12:38:29.944 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.942+0000 7fc9f585a0c0 20 remove_watcher() i=2 2020-11-09T12:38:29.949 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.950+0000 7fc9f585a0c0 20 remove_watcher() i=3 2020-11-09T12:38:29.951 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.950+0000 7fc9f585a0c0 20 remove_watcher() i=4 2020-11-09T12:38:29.954 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.954+0000 7fc9f585a0c0 20 remove_watcher() i=5 2020-11-09T12:38:29.957 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.958+0000 7fc9f585a0c0 20 remove_watcher() i=6 2020-11-09T12:38:29.961 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.958+0000 7fc9f585a0c0 20 remove_watcher() i=7 2020-11-09T12:38:29.962 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:29.962+0000 7fc9b27e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T12:38:29.991 DEBUG:tasks.util.rgw: json result: [] 2020-11-09T12:38:29.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T12:38:29.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T12:38:29.993 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-09T12:38:30.067 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:30.067 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:30.074 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.074+0000 7fb2716d80c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T12:38:30.074 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.074+0000 7fb2716d80c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T12:38:30.080 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.078+0000 7fb1a67fc700 20 reqs_thread_entry: start 2020-11-09T12:38:30.081 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.078+0000 7fb2716d80c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.081 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.082+0000 7fb2716d80c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:30.081 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.082+0000 7fb2716d80c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.083 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.082+0000 7fb2716d80c0 20 rados_obj.operate() r=0 bl.length=873 2020-11-09T12:38:30.083 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.082+0000 7fb2716d80c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.083 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.082+0000 7fb2716d80c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:30.084 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.082+0000 7fb2716d80c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.084 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.082+0000 7fb2716d80c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:30.084 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.082+0000 7fb2716d80c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.085 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.086+0000 7fb2716d80c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T12:38:30.093 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:38:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "e6c9450e-d772-4559-a326-d447647d396f", 2020-11-09T12:38:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "default", 2020-11-09T12:38:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "domain_root": "default.rgw.meta:root", 2020-11-09T12:38:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "control_pool": "default.rgw.control", 2020-11-09T12:38:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-09T12:38:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-09T12:38:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "log_pool": "default.rgw.log", 2020-11-09T12:38:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-09T12:38:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-09T12:38:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-09T12:38:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-09T12:38:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-09T12:38:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-09T12:38:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-09T12:38:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-09T12:38:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: "otp_pool": "default.rgw.otp", 2020-11-09T12:38:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: "system_key": { 2020-11-09T12:38:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: "access_key": "", 2020-11-09T12:38:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: "secret_key": "" 2020-11-09T12:38:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:38:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: "placement_pools": [ 2020-11-09T12:38:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:38:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "key": "default-placement", 2020-11-09T12:38:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:38:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-09T12:38:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "storage_classes": { 2020-11-09T12:38:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "STANDARD": { 2020-11-09T12:38:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-09T12:38:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "compression_type": "random" 2020-11-09T12:38:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:38:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:38:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-09T12:38:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: "index_type": 0 2020-11-09T12:38:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:38:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:38:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:38:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_id": "", 2020-11-09T12:38:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-09T12:38:30.101 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:38:30.102 DEBUG:tasks.util.rgw: json result: {'id': 'e6c9450e-d772-4559-a326-d447647d396f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-09T12:38:30.103 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-11-09T12:38:30.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-11-09T12:38:30.104 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-11-09T12:38:30.178 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:30.179 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:30.186 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.186+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.186 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.186+0000 7f216c2f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:30.186 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.186+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.187 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.186+0000 7f216c2f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:30.187 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.186+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.187 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.186+0000 7f216c2f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:30.188 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.186+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.188 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.186+0000 7f216c2f40c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:30.206 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.206+0000 7f216c2f40c0 20 RGWRados::pool_iterate: got zone_info.e6c9450e-d772-4559-a326-d447647d396f 2020-11-09T12:38:30.206 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.206+0000 7f216c2f40c0 20 RGWRados::pool_iterate: got zonegroup_info.7dbd9f12-3414-4b86-8000-367c587cdef0 2020-11-09T12:38:30.207 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.206+0000 7f216c2f40c0 20 RGWRados::pool_iterate: got zone_names.default 2020-11-09T12:38:30.207 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.206+0000 7f216c2f40c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-11-09T12:38:30.207 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.206+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.207 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.206+0000 7f216c2f40c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:30.207 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.206+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.210 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.210+0000 7f216c2f40c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-09T12:38:30.211 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.210+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.212 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.210+0000 7f216c2f40c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:30.212 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.210+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.212 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.210+0000 7f216c2f40c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T12:38:30.212 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.210+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.213 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.210+0000 7f216c2f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:30.213 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.210+0000 7f216c2f40c0 10 cannot find current period zonegroup using local zonegroup 2020-11-09T12:38:30.213 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.210+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.213 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.214+0000 7f216c2f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:30.214 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.214+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.214 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.214+0000 7f216c2f40c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:30.214 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.214+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.215 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.214+0000 7f216c2f40c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T12:38:30.215 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.214+0000 7f216c2f40c0 20 zonegroup default 2020-11-09T12:38:30.215 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.214+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.215 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.214+0000 7f216c2f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:30.216 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.214+0000 7f216c2f40c0 10 Cannot find current period zone using local zone 2020-11-09T12:38:30.216 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.214+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.216 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.214+0000 7f216c2f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:30.216 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.214+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.217 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.214+0000 7f216c2f40c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:30.217 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.214+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.217 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.218+0000 7f216c2f40c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-09T12:38:30.217 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.218+0000 7f216c2f40c0 20 zone default found 2020-11-09T12:38:30.217 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.218+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.218 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.218+0000 7f216c2f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:30.218 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.218+0000 7f216c2f40c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.218 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.218+0000 7f216c2f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:30.218 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.218+0000 7f216c2f40c0 20 started sync module instance, tier type = 2020-11-09T12:38:30.219 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.218+0000 7f216c2f40c0 20 started zone id=e6c9450e-d772-4559-a326-d447647d396f (name=default) with tier type = 2020-11-09T12:38:30.248 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.246+0000 7f216c2f40c0 20 add_watcher() i=0 2020-11-09T12:38:30.249 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.246+0000 7f216c2f40c0 20 add_watcher() i=1 2020-11-09T12:38:30.249 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.246+0000 7f216c2f40c0 20 add_watcher() i=2 2020-11-09T12:38:30.249 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.246+0000 7f216c2f40c0 20 add_watcher() i=3 2020-11-09T12:38:30.249 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.246+0000 7f216c2f40c0 20 add_watcher() i=4 2020-11-09T12:38:30.249 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.246+0000 7f216c2f40c0 20 add_watcher() i=5 2020-11-09T12:38:30.252 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.250+0000 7f216c2f40c0 20 add_watcher() i=6 2020-11-09T12:38:30.252 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.250+0000 7f216c2f40c0 20 add_watcher() i=7 2020-11-09T12:38:30.252 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.250+0000 7f216c2f40c0 2 all 8 watchers are set, enabling cache 2020-11-09T12:38:30.253 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.254+0000 7f216c2f40c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:30.254 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.254+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T12:38:30.254 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.254+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T12:38:30.255 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.254+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T12:38:30.255 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.254+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T12:38:30.255 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.254+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T12:38:30.256 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.254+0000 7f216c2f40c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:30.256 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.254+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T12:38:30.257 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.254+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T12:38:30.257 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.258+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T12:38:30.257 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.258+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T12:38:30.258 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.258+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T12:38:30.258 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.258+0000 7f216c2f40c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:30.258 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.258+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T12:38:30.259 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.258+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T12:38:30.259 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.258+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T12:38:30.259 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.258+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T12:38:30.260 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.258+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T12:38:30.260 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.258+0000 7f216c2f40c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:30.260 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.258+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-09T12:38:30.260 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.258+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-09T12:38:30.261 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.262+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-09T12:38:30.261 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.262+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-09T12:38:30.261 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.262+0000 7f216c2f40c0 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-09T12:38:30.262 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.262+0000 7f216c2f40c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:30.264 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.262+0000 7f216c2f40c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:30.264 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.262+0000 7f216c2f40c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:30.267 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.266+0000 7f216c2f40c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:30.268 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.266+0000 7f216c2f40c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:30.271 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.270+0000 7f216c2f40c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:30.271 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.270+0000 7f216c2f40c0 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-09T12:38:30.275 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.274+0000 7f216c2f40c0 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-09T12:38:30.276 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.274+0000 7f21297e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T12:38:30.276 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.274+0000 7f216c2f40c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T12:38:30.276 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.274+0000 7f216c2f40c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T12:38:30.373 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.370+0000 7f2097fff700 20 reqs_thread_entry: start 2020-11-09T12:38:30.373 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.370+0000 7f2095ffb700 20 reqs_thread_entry: start 2020-11-09T12:38:30.373 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.370+0000 7f216c2f40c0 20 init_complete bucket index max shards: 11 2020-11-09T12:38:30.373 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.370+0000 7f216c2f40c0 10 Started notification manager with: 1 workers 2020-11-09T12:38:30.374 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.370+0000 7f2083fff700 20 INFO: next queues processing will happen at: Mon Nov 9 12:39:00 2020 2020-11-09T12:38:30.374 INFO:teuthology.orchestra.run.smithi190.stderr: 2020-11-09T12:38:30.374 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.374+0000 7f20837fe700 20 reqs_thread_entry: start 2020-11-09T12:38:30.402 INFO:teuthology.orchestra.run.smithi190.stdout:[] 2020-11-09T12:38:30.403 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.402+0000 7f216c2f40c0 20 remove_watcher() i=0 2020-11-09T12:38:30.403 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.402+0000 7f216c2f40c0 2 removed watcher, disabling cache 2020-11-09T12:38:30.404 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.402+0000 7f216c2f40c0 20 remove_watcher() i=1 2020-11-09T12:38:30.407 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.406+0000 7f216c2f40c0 20 remove_watcher() i=2 2020-11-09T12:38:30.410 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.410+0000 7f216c2f40c0 20 remove_watcher() i=3 2020-11-09T12:38:30.414 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.414+0000 7f216c2f40c0 20 remove_watcher() i=4 2020-11-09T12:38:30.416 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.414+0000 7f216c2f40c0 20 remove_watcher() i=5 2020-11-09T12:38:30.419 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.418+0000 7f216c2f40c0 20 remove_watcher() i=6 2020-11-09T12:38:30.422 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.422+0000 7f216c2f40c0 20 remove_watcher() i=7 2020-11-09T12:38:30.423 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.422+0000 7f21297e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-09T12:38:30.428 DEBUG:tasks.util.rgw: json result: [] 2020-11-09T12:38:30.429 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T12:38:30.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-09T12:38:30.429 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-09T12:38:30.510 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:30.510 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:30.523 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.522+0000 7f03601980c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-09T12:38:30.523 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.522+0000 7f03601980c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-09T12:38:30.530 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.530+0000 7f03601980c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.530 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.530+0000 7f02b57fa700 20 reqs_thread_entry: start 2020-11-09T12:38:30.531 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.530+0000 7f03601980c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:30.532 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.530+0000 7f03601980c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.534+0000 7f03601980c0 20 rados_obj.operate() r=0 bl.length=889 2020-11-09T12:38:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.534+0000 7f03601980c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.534+0000 7f03601980c0 20 rados_obj.operate() r=-2 bl.length=0 2020-11-09T12:38:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.534+0000 7f03601980c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.534+0000 7f03601980c0 20 rados_obj.operate() r=0 bl.length=46 2020-11-09T12:38:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.534+0000 7f03601980c0 20 rados->read ofs=0 len=0 2020-11-09T12:38:30.536 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:30.534+0000 7f03601980c0 20 rados_obj.operate() r=0 bl.length=358 2020-11-09T12:38:30.544 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:38:30.544 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "e6c9450e-d772-4559-a326-d447647d396f", 2020-11-09T12:38:30.545 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "default", 2020-11-09T12:38:30.545 INFO:teuthology.orchestra.run.smithi190.stdout: "domain_root": "default.rgw.meta:root", 2020-11-09T12:38:30.545 INFO:teuthology.orchestra.run.smithi190.stdout: "control_pool": "default.rgw.control", 2020-11-09T12:38:30.545 INFO:teuthology.orchestra.run.smithi190.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-09T12:38:30.545 INFO:teuthology.orchestra.run.smithi190.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-09T12:38:30.545 INFO:teuthology.orchestra.run.smithi190.stdout: "log_pool": "default.rgw.log", 2020-11-09T12:38:30.545 INFO:teuthology.orchestra.run.smithi190.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-09T12:38:30.546 INFO:teuthology.orchestra.run.smithi190.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-09T12:38:30.546 INFO:teuthology.orchestra.run.smithi190.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-09T12:38:30.546 INFO:teuthology.orchestra.run.smithi190.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-09T12:38:30.546 INFO:teuthology.orchestra.run.smithi190.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-09T12:38:30.546 INFO:teuthology.orchestra.run.smithi190.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-09T12:38:30.546 INFO:teuthology.orchestra.run.smithi190.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-09T12:38:30.547 INFO:teuthology.orchestra.run.smithi190.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-09T12:38:30.547 INFO:teuthology.orchestra.run.smithi190.stdout: "otp_pool": "default.rgw.otp", 2020-11-09T12:38:30.547 INFO:teuthology.orchestra.run.smithi190.stdout: "system_key": { 2020-11-09T12:38:30.547 INFO:teuthology.orchestra.run.smithi190.stdout: "access_key": "", 2020-11-09T12:38:30.547 INFO:teuthology.orchestra.run.smithi190.stdout: "secret_key": "" 2020-11-09T12:38:30.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:38:30.547 INFO:teuthology.orchestra.run.smithi190.stdout: "placement_pools": [ 2020-11-09T12:38:30.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:38:30.548 INFO:teuthology.orchestra.run.smithi190.stdout: "key": "default-placement", 2020-11-09T12:38:30.548 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:38:30.548 INFO:teuthology.orchestra.run.smithi190.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-09T12:38:30.548 INFO:teuthology.orchestra.run.smithi190.stdout: "storage_classes": { 2020-11-09T12:38:30.548 INFO:teuthology.orchestra.run.smithi190.stdout: "STANDARD": { 2020-11-09T12:38:30.548 INFO:teuthology.orchestra.run.smithi190.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-09T12:38:30.549 INFO:teuthology.orchestra.run.smithi190.stdout: "compression_type": "random" 2020-11-09T12:38:30.549 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:38:30.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:38:30.549 INFO:teuthology.orchestra.run.smithi190.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-09T12:38:30.549 INFO:teuthology.orchestra.run.smithi190.stdout: "index_type": 0 2020-11-09T12:38:30.549 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:38:30.549 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:38:30.550 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:38:30.550 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_id": "", 2020-11-09T12:38:30.550 INFO:teuthology.orchestra.run.smithi190.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-09T12:38:30.550 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:38:30.551 DEBUG:tasks.util.rgw: json result: {'id': 'e6c9450e-d772-4559-a326-d447647d396f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-09T12:38:30.551 INFO:tasks.rgw:Starting rgw... 2020-11-09T12:38:30.551 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2020-11-09T12:38:30.552 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-11-09T12:38:30.552 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-11-09T12:38:30.553 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-11-09T12:38:30.555 INFO:tasks.rgw.c1.client.0:Started 2020-11-09T12:38:30.555 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2020-11-09T12:38:30.556 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-11-09T12:38:30.556 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-09T12:38:30.556 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-09T12:38:30.558 INFO:tasks.rgw.c1.client.1:Started 2020-11-09T12:38:30.558 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2020-11-09T12:38:30.559 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-11-09T12:38:30.559 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-11-09T12:38:30.559 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-11-09T12:38:30.594 INFO:tasks.rgw.c2.client.0:Started 2020-11-09T12:38:30.595 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2020-11-09T12:38:30.595 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-11-09T12:38:30.595 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-11-09T12:38:30.596 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-11-09T12:38:30.598 INFO:tasks.rgw.c2.client.1:Started 2020-11-09T12:38:30.598 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi164.front.sepia.ceph.com:8000/ 2020-11-09T12:38:30.599 INFO:teuthology.orchestra.run.smithi164:> curl http://smithi164.front.sepia.ceph.com:8000/ 2020-11-09T12:38:30.625 INFO:teuthology.orchestra.run.smithi164.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T12:38:30.625 INFO:teuthology.orchestra.run.smithi164.stderr: Dload Upload Total Spent Left Speed 2020-11-09T12:38:30.628 INFO:teuthology.orchestra.run.smithi164.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi164.front.sepia.ceph.com port 8000: Connection refused 2020-11-09T12:38:30.629 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-09T12:38:31.631 INFO:teuthology.orchestra.run.smithi164:> curl http://smithi164.front.sepia.ceph.com:8000/ 2020-11-09T12:38:31.652 INFO:teuthology.orchestra.run.smithi164.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T12:38:31.653 INFO:teuthology.orchestra.run.smithi164.stderr: Dload Upload Total Spent Left Speed 2020-11-09T12:38:31.660 INFO:teuthology.orchestra.run.smithi164.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-09T12:38:31.662 INFO:teuthology.orchestra.run.smithi164.stdout:anonymous 2020-11-09T12:38:31.663 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi164.front.sepia.ceph.com:8001/ 2020-11-09T12:38:31.668 INFO:teuthology.orchestra.run.smithi164:> curl http://smithi164.front.sepia.ceph.com:8001/ 2020-11-09T12:38:31.728 INFO:teuthology.orchestra.run.smithi164.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T12:38:31.729 INFO:teuthology.orchestra.run.smithi164.stderr: Dload Upload Total Spent Left Speed 2020-11-09T12:38:31.731 INFO:teuthology.orchestra.run.smithi164.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2020-11-09T12:38:31.731 INFO:teuthology.orchestra.run.smithi164.stdout:anonymous 2020-11-09T12:38:31.732 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi190.front.sepia.ceph.com:8000/ 2020-11-09T12:38:31.733 INFO:teuthology.orchestra.run.smithi190:> curl http://smithi190.front.sepia.ceph.com:8000/ 2020-11-09T12:38:31.923 INFO:teuthology.orchestra.run.smithi190.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T12:38:31.923 INFO:teuthology.orchestra.run.smithi190.stderr: Dload Upload Total Spent Left Speed 2020-11-09T12:38:31.930 INFO:teuthology.orchestra.run.smithi190.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 26750 0 --:--:-- --:--:-- --:--:-- 30571 2020-11-09T12:38:31.931 INFO:teuthology.orchestra.run.smithi190.stdout:anonymous 2020-11-09T12:38:31.931 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi190.front.sepia.ceph.com:8001/ 2020-11-09T12:38:31.933 INFO:teuthology.orchestra.run.smithi190:> curl http://smithi190.front.sepia.ceph.com:8001/ 2020-11-09T12:38:31.951 INFO:teuthology.orchestra.run.smithi190.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T12:38:31.951 INFO:teuthology.orchestra.run.smithi190.stderr: Dload Upload Total Spent Left Speed 2020-11-09T12:38:31.958 INFO:teuthology.orchestra.run.smithi190.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2020-11-09T12:38:31.959 INFO:teuthology.orchestra.run.smithi190.stdout:anonymous 2020-11-09T12:38:31.960 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-11-09T12:38:32.362 INFO:tasks.rgw_multisite:creating realm.. 2020-11-09T12:38:32.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:32.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:32.367 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T12:38:32.403 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:32.403 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:32.459 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:38:32.459 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:38:32.460 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "test-realm", 2020-11-09T12:38:32.460 INFO:teuthology.orchestra.run.smithi164.stdout: "current_period": "411b4533-24c9-4015-aaf1-5ab40f2fe3c0", 2020-11-09T12:38:32.460 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1 2020-11-09T12:38:32.460 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:38:32.461 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-11-09T12:38:32.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:32.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:32.462 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T12:38:32.539 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:32.539 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:32.564 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:38:32.559+0000 7f5e344800c0 -1 old format 2020-11-09T12:38:32.579 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:38:32.580 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "test-zonegroup", 2020-11-09T12:38:32.580 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "test-zonegroup", 2020-11-09T12:38:32.580 INFO:teuthology.orchestra.run.smithi164.stdout: "api_name": "", 2020-11-09T12:38:32.580 INFO:teuthology.orchestra.run.smithi164.stdout: "is_master": "true", 2020-11-09T12:38:32.580 INFO:teuthology.orchestra.run.smithi164.stdout: "endpoints": [ 2020-11-09T12:38:32.581 INFO:teuthology.orchestra.run.smithi164.stdout: "http://smithi164.front.sepia.ceph.com:8000" 2020-11-09T12:38:32.581 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:38:32.581 INFO:teuthology.orchestra.run.smithi164.stdout: "hostnames": [], 2020-11-09T12:38:32.581 INFO:teuthology.orchestra.run.smithi164.stdout: "hostnames_s3website": [], 2020-11-09T12:38:32.581 INFO:teuthology.orchestra.run.smithi164.stdout: "master_zone": "", 2020-11-09T12:38:32.581 INFO:teuthology.orchestra.run.smithi164.stdout: "zones": [], 2020-11-09T12:38:32.582 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_targets": [], 2020-11-09T12:38:32.583 INFO:teuthology.orchestra.run.smithi164.stdout: "default_placement": "", 2020-11-09T12:38:32.583 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:38:32.583 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_policy": { 2020-11-09T12:38:32.583 INFO:teuthology.orchestra.run.smithi164.stdout: "groups": [] 2020-11-09T12:38:32.583 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:32.584 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:38:32.585 INFO:tasks.rgw_multisite:creating master zone.. 2020-11-09T12:38:32.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--endpoints', 'http://smithi164.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:32.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--endpoints', 'http://smithi164.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:32.587 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key JX6ROEU3B1OMPZYL --secret GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme --endpoints http://smithi164.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T12:38:32.673 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:32.673 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:32.748 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:38:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:38:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "test-zone1", 2020-11-09T12:38:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-09T12:38:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-09T12:38:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-09T12:38:32.749 INFO:teuthology.orchestra.run.smithi164.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-09T12:38:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-09T12:38:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-09T12:38:32.750 INFO:teuthology.orchestra.run.smithi164.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-09T12:38:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-09T12:38:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-09T12:38:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-09T12:38:32.751 INFO:teuthology.orchestra.run.smithi164.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-09T12:38:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-09T12:38:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-09T12:38:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-09T12:38:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "system_key": { 2020-11-09T12:38:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "access_key": "JX6ROEU3B1OMPZYL", 2020-11-09T12:38:32.752 INFO:teuthology.orchestra.run.smithi164.stdout: "secret_key": "GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme" 2020-11-09T12:38:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:38:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_pools": [ 2020-11-09T12:38:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:38:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "key": "default-placement", 2020-11-09T12:38:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:38:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-09T12:38:32.753 INFO:teuthology.orchestra.run.smithi164.stdout: "storage_classes": { 2020-11-09T12:38:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: "STANDARD": { 2020-11-09T12:38:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2020-11-09T12:38:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:38:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-09T12:38:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: "index_type": 0 2020-11-09T12:38:32.754 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:38:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:38:32.755 INFO:teuthology.orchestra.run.smithi164.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-09T12:38:32.755 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:38:32.756 INFO:teuthology.orchestra.run.smithi164:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-11-09T12:38:34.007 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-11-09T12:38:34.022 INFO:teuthology.orchestra.run.smithi164:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-11-09T12:38:35.022 INFO:teuthology.orchestra.run.smithi164.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-11-09T12:38:35.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:35.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:35.050 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T12:38:35.094 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:35.095 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:35.115 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:38:35.115 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:38:35.116 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "test-zone1", 2020-11-09T12:38:35.116 INFO:teuthology.orchestra.run.smithi164.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-09T12:38:35.116 INFO:teuthology.orchestra.run.smithi164.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-09T12:38:35.116 INFO:teuthology.orchestra.run.smithi164.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-09T12:38:35.116 INFO:teuthology.orchestra.run.smithi164.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-09T12:38:35.116 INFO:teuthology.orchestra.run.smithi164.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-09T12:38:35.116 INFO:teuthology.orchestra.run.smithi164.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-09T12:38:35.117 INFO:teuthology.orchestra.run.smithi164.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-09T12:38:35.117 INFO:teuthology.orchestra.run.smithi164.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-09T12:38:35.117 INFO:teuthology.orchestra.run.smithi164.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-09T12:38:35.117 INFO:teuthology.orchestra.run.smithi164.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-09T12:38:35.117 INFO:teuthology.orchestra.run.smithi164.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-09T12:38:35.117 INFO:teuthology.orchestra.run.smithi164.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-09T12:38:35.117 INFO:teuthology.orchestra.run.smithi164.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-09T12:38:35.118 INFO:teuthology.orchestra.run.smithi164.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-09T12:38:35.118 INFO:teuthology.orchestra.run.smithi164.stdout: "system_key": { 2020-11-09T12:38:35.118 INFO:teuthology.orchestra.run.smithi164.stdout: "access_key": "JX6ROEU3B1OMPZYL", 2020-11-09T12:38:35.118 INFO:teuthology.orchestra.run.smithi164.stdout: "secret_key": "GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme" 2020-11-09T12:38:35.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:38:35.118 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_pools": [ 2020-11-09T12:38:35.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:38:35.119 INFO:teuthology.orchestra.run.smithi164.stdout: "key": "default-placement", 2020-11-09T12:38:35.119 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:38:35.119 INFO:teuthology.orchestra.run.smithi164.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-09T12:38:35.119 INFO:teuthology.orchestra.run.smithi164.stdout: "storage_classes": { 2020-11-09T12:38:35.119 INFO:teuthology.orchestra.run.smithi164.stdout: "STANDARD": { 2020-11-09T12:38:35.119 INFO:teuthology.orchestra.run.smithi164.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-11-09T12:38:35.120 INFO:teuthology.orchestra.run.smithi164.stdout: "compression_type": "random" 2020-11-09T12:38:35.120 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:35.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:38:35.120 INFO:teuthology.orchestra.run.smithi164.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-09T12:38:35.120 INFO:teuthology.orchestra.run.smithi164.stdout: "index_type": 0 2020-11-09T12:38:35.120 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:35.120 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:35.121 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:38:35.121 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "", 2020-11-09T12:38:35.121 INFO:teuthology.orchestra.run.smithi164.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-09T12:38:35.121 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:38:35.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:35.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:35.123 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T12:38:35.205 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:35.205 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:43.273 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:38:43.273 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:38:43.273 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2020-11-09T12:38:43.273 INFO:teuthology.orchestra.run.smithi164.stdout: "predecessor_uuid": "411b4533-24c9-4015-aaf1-5ab40f2fe3c0", 2020-11-09T12:38:43.274 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": [], 2020-11-09T12:38:43.274 INFO:teuthology.orchestra.run.smithi164.stdout: "period_map": { 2020-11-09T12:38:43.274 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:38:43.274 INFO:teuthology.orchestra.run.smithi164.stdout: "zonegroups": [ 2020-11-09T12:38:43.274 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:38:43.274 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "test-zonegroup", 2020-11-09T12:38:43.274 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "test-zonegroup", 2020-11-09T12:38:43.275 INFO:teuthology.orchestra.run.smithi164.stdout: "api_name": "", 2020-11-09T12:38:43.275 INFO:teuthology.orchestra.run.smithi164.stdout: "is_master": "true", 2020-11-09T12:38:43.275 INFO:teuthology.orchestra.run.smithi164.stdout: "endpoints": [ 2020-11-09T12:38:43.275 INFO:teuthology.orchestra.run.smithi164.stdout: "http://smithi164.front.sepia.ceph.com:8000" 2020-11-09T12:38:43.275 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:38:43.275 INFO:teuthology.orchestra.run.smithi164.stdout: "hostnames": [], 2020-11-09T12:38:43.275 INFO:teuthology.orchestra.run.smithi164.stdout: "hostnames_s3website": [], 2020-11-09T12:38:43.276 INFO:teuthology.orchestra.run.smithi164.stdout: "master_zone": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:38:43.276 INFO:teuthology.orchestra.run.smithi164.stdout: "zones": [ 2020-11-09T12:38:43.276 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:38:43.276 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:38:43.276 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "test-zone1", 2020-11-09T12:38:43.276 INFO:teuthology.orchestra.run.smithi164.stdout: "endpoints": [ 2020-11-09T12:38:43.276 INFO:teuthology.orchestra.run.smithi164.stdout: "http://smithi164.front.sepia.ceph.com:8000" 2020-11-09T12:38:43.277 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:38:43.277 INFO:teuthology.orchestra.run.smithi164.stdout: "log_meta": "false", 2020-11-09T12:38:43.277 INFO:teuthology.orchestra.run.smithi164.stdout: "log_data": "false", 2020-11-09T12:38:43.277 INFO:teuthology.orchestra.run.smithi164.stdout: "bucket_index_max_shards": 11, 2020-11-09T12:38:43.277 INFO:teuthology.orchestra.run.smithi164.stdout: "read_only": "false", 2020-11-09T12:38:43.277 INFO:teuthology.orchestra.run.smithi164.stdout: "tier_type": "", 2020-11-09T12:38:43.277 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_from_all": "true", 2020-11-09T12:38:43.278 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_from": [], 2020-11-09T12:38:43.278 INFO:teuthology.orchestra.run.smithi164.stdout: "redirect_zone": "" 2020-11-09T12:38:43.278 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:43.278 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:38:43.278 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_targets": [ 2020-11-09T12:38:43.278 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:38:43.278 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "default-placement", 2020-11-09T12:38:43.278 INFO:teuthology.orchestra.run.smithi164.stdout: "tags": [], 2020-11-09T12:38:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: "storage_classes": [ 2020-11-09T12:38:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: "STANDARD" 2020-11-09T12:38:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:38:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:38:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: "default_placement": "default-placement", 2020-11-09T12:38:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:38:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_policy": { 2020-11-09T12:38:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "groups": [] 2020-11-09T12:38:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:38:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "short_zone_ids": [ 2020-11-09T12:38:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:38:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "key": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:38:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 1014254849 2020-11-09T12:38:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:38:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:38:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T12:38:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "master_zone": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:38:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "period_config": { 2020-11-09T12:38:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "bucket_quota": { 2020-11-09T12:38:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "enabled": false, 2020-11-09T12:38:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "check_on_raw": false, 2020-11-09T12:38:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size": -1, 2020-11-09T12:38:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size_kb": 0, 2020-11-09T12:38:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "max_objects": -1 2020-11-09T12:38:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:38:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "user_quota": { 2020-11-09T12:38:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "enabled": false, 2020-11-09T12:38:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "check_on_raw": false, 2020-11-09T12:38:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size": -1, 2020-11-09T12:38:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size_kb": 0, 2020-11-09T12:38:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "max_objects": -1 2020-11-09T12:38:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:38:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:38:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_name": "test-realm", 2020-11-09T12:38:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:38:43.284 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:38:43.285 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-11-09T12:38:43.285 INFO:teuthology.orchestra.run:waiting for 300 2020-11-09T12:38:43.329 INFO:tasks.rgw.c1.client.0.smithi164.stdout:2020-11-09T12:38:43.327+0000 7f7482ffd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 14493) UID: 0 2020-11-09T12:38:43.329 INFO:tasks.rgw.c1.client.0.smithi164.stdout:2020-11-09T12:38:43.327+0000 7f749e7ed100 -1 shutting down 2020-11-09T12:38:46.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2020-11-09T12:38:49.387 INFO:tasks.rgw.c1.client.0:Stopped 2020-11-09T12:38:49.387 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-11-09T12:38:49.388 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-11-09T12:38:49.390 INFO:tasks.rgw.c1.client.0:Started 2020-11-09T12:38:49.391 INFO:teuthology.orchestra.run.smithi164:> curl http://smithi164.front.sepia.ceph.com:8000 2020-11-09T12:38:49.407 INFO:teuthology.orchestra.run.smithi164.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T12:38:49.408 INFO:teuthology.orchestra.run.smithi164.stderr: Dload Upload Total Spent Left Speed 2020-11-09T12:38:49.413 INFO:teuthology.orchestra.run.smithi164.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi164.front.sepia.ceph.com port 8000: Connection refused 2020-11-09T12:38:49.414 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-09T12:38:50.415 INFO:teuthology.orchestra.run.smithi164:> curl http://smithi164.front.sepia.ceph.com:8000 2020-11-09T12:38:50.431 INFO:teuthology.orchestra.run.smithi164.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T12:38:50.432 INFO:teuthology.orchestra.run.smithi164.stderr: Dload Upload Total Spent Left Speed 2020-11-09T12:38:50.436 INFO:teuthology.orchestra.run.smithi164.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-09T12:38:50.436 INFO:teuthology.orchestra.run.smithi164.stdout:anonymous 2020-11-09T12:38:50.437 INFO:tasks.rgw_multisite:creating admin user.. 2020-11-09T12:38:50.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:50.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:50.439 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key JX6ROEU3B1OMPZYL --secret GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T12:38:50.532 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:50.533 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:50.765 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:38:50.766 INFO:teuthology.orchestra.run.smithi164.stdout: "user_id": "realm-admin", 2020-11-09T12:38:50.766 INFO:teuthology.orchestra.run.smithi164.stdout: "display_name": "Realm Admin", 2020-11-09T12:38:50.766 INFO:teuthology.orchestra.run.smithi164.stdout: "email": "", 2020-11-09T12:38:50.766 INFO:teuthology.orchestra.run.smithi164.stdout: "suspended": 0, 2020-11-09T12:38:50.766 INFO:teuthology.orchestra.run.smithi164.stdout: "max_buckets": 1000, 2020-11-09T12:38:50.766 INFO:teuthology.orchestra.run.smithi164.stdout: "subusers": [], 2020-11-09T12:38:50.767 INFO:teuthology.orchestra.run.smithi164.stdout: "keys": [ 2020-11-09T12:38:50.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:38:50.767 INFO:teuthology.orchestra.run.smithi164.stdout: "user": "realm-admin", 2020-11-09T12:38:50.767 INFO:teuthology.orchestra.run.smithi164.stdout: "access_key": "JX6ROEU3B1OMPZYL", 2020-11-09T12:38:50.767 INFO:teuthology.orchestra.run.smithi164.stdout: "secret_key": "GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme" 2020-11-09T12:38:50.767 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:38:50.767 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:38:50.768 INFO:teuthology.orchestra.run.smithi164.stdout: "swift_keys": [], 2020-11-09T12:38:50.768 INFO:teuthology.orchestra.run.smithi164.stdout: "caps": [], 2020-11-09T12:38:50.768 INFO:teuthology.orchestra.run.smithi164.stdout: "op_mask": "read, write, delete", 2020-11-09T12:38:50.768 INFO:teuthology.orchestra.run.smithi164.stdout: "system": "true", 2020-11-09T12:38:50.768 INFO:teuthology.orchestra.run.smithi164.stdout: "default_placement": "", 2020-11-09T12:38:50.768 INFO:teuthology.orchestra.run.smithi164.stdout: "default_storage_class": "", 2020-11-09T12:38:50.768 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_tags": [], 2020-11-09T12:38:50.769 INFO:teuthology.orchestra.run.smithi164.stdout: "bucket_quota": { 2020-11-09T12:38:50.769 INFO:teuthology.orchestra.run.smithi164.stdout: "enabled": false, 2020-11-09T12:38:50.769 INFO:teuthology.orchestra.run.smithi164.stdout: "check_on_raw": false, 2020-11-09T12:38:50.769 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size": -1, 2020-11-09T12:38:50.769 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size_kb": 0, 2020-11-09T12:38:50.769 INFO:teuthology.orchestra.run.smithi164.stdout: "max_objects": -1 2020-11-09T12:38:50.769 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:38:50.770 INFO:teuthology.orchestra.run.smithi164.stdout: "user_quota": { 2020-11-09T12:38:50.770 INFO:teuthology.orchestra.run.smithi164.stdout: "enabled": false, 2020-11-09T12:38:50.770 INFO:teuthology.orchestra.run.smithi164.stdout: "check_on_raw": false, 2020-11-09T12:38:50.770 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size": -1, 2020-11-09T12:38:50.770 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size_kb": 0, 2020-11-09T12:38:50.770 INFO:teuthology.orchestra.run.smithi164.stdout: "max_objects": -1 2020-11-09T12:38:50.770 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:38:50.770 INFO:teuthology.orchestra.run.smithi164.stdout: "temp_url_keys": [], 2020-11-09T12:38:50.771 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "rgw", 2020-11-09T12:38:50.771 INFO:teuthology.orchestra.run.smithi164.stdout: "mfa_ids": [] 2020-11-09T12:38:50.771 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:38:50.771 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:38:50.799 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-11-09T12:38:50.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi164.front.sepia.ceph.com:8000', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:50.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi164.front.sepia.ceph.com:8000', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:50.810 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi164.front.sepia.ceph.com:8000 --access-key JX6ROEU3B1OMPZYL --secret GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T12:38:50.853 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:50.854 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:50.973 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:38:50.973 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:38:50.973 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "test-realm", 2020-11-09T12:38:50.973 INFO:teuthology.orchestra.run.smithi190.stdout: "current_period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:38:50.974 INFO:teuthology.orchestra.run.smithi190.stdout: "epoch": 2 2020-11-09T12:38:50.974 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:38:50.975 INFO:tasks.rgw_multisite:creating zone.. 2020-11-09T12:38:50.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--endpoints', 'http://smithi190.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:50.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--endpoints', 'http://smithi190.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:50.976 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key JX6ROEU3B1OMPZYL --secret GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme --endpoints http://smithi190.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T12:38:51.062 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:51.062 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:51.125 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:51.122+0000 7fb3b27c20c0 0 failed reading obj info from .rgw.root:zone_info.6dfaac43-4458-47cb-abae-96e6cb987c71: (2) No such file or directory 2020-11-09T12:38:51.126 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:38:51.122+0000 7fb3b27c20c0 0 WARNING: could not read zone params for zone id=6dfaac43-4458-47cb-abae-96e6cb987c71 name=test-zone1 2020-11-09T12:38:51.136 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:38:51.137 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "c104565e-7c15-4c5e-9b2c-908de1973930", 2020-11-09T12:38:51.137 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "test-zone2", 2020-11-09T12:38:51.137 INFO:teuthology.orchestra.run.smithi190.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-09T12:38:51.137 INFO:teuthology.orchestra.run.smithi190.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-09T12:38:51.137 INFO:teuthology.orchestra.run.smithi190.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-09T12:38:51.137 INFO:teuthology.orchestra.run.smithi190.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-09T12:38:51.138 INFO:teuthology.orchestra.run.smithi190.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-09T12:38:51.138 INFO:teuthology.orchestra.run.smithi190.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-09T12:38:51.138 INFO:teuthology.orchestra.run.smithi190.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-09T12:38:51.138 INFO:teuthology.orchestra.run.smithi190.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-09T12:38:51.138 INFO:teuthology.orchestra.run.smithi190.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-09T12:38:51.138 INFO:teuthology.orchestra.run.smithi190.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-09T12:38:51.139 INFO:teuthology.orchestra.run.smithi190.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-09T12:38:51.139 INFO:teuthology.orchestra.run.smithi190.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-09T12:38:51.139 INFO:teuthology.orchestra.run.smithi190.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-09T12:38:51.139 INFO:teuthology.orchestra.run.smithi190.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-09T12:38:51.139 INFO:teuthology.orchestra.run.smithi190.stdout: "system_key": { 2020-11-09T12:38:51.139 INFO:teuthology.orchestra.run.smithi190.stdout: "access_key": "JX6ROEU3B1OMPZYL", 2020-11-09T12:38:51.140 INFO:teuthology.orchestra.run.smithi190.stdout: "secret_key": "GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme" 2020-11-09T12:38:51.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:38:51.140 INFO:teuthology.orchestra.run.smithi190.stdout: "placement_pools": [ 2020-11-09T12:38:51.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:38:51.140 INFO:teuthology.orchestra.run.smithi190.stdout: "key": "default-placement", 2020-11-09T12:38:51.140 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:38:51.140 INFO:teuthology.orchestra.run.smithi190.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-09T12:38:51.141 INFO:teuthology.orchestra.run.smithi190.stdout: "storage_classes": { 2020-11-09T12:38:51.141 INFO:teuthology.orchestra.run.smithi190.stdout: "STANDARD": { 2020-11-09T12:38:51.141 INFO:teuthology.orchestra.run.smithi190.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2020-11-09T12:38:51.141 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:38:51.142 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:38:51.142 INFO:teuthology.orchestra.run.smithi190.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-09T12:38:51.142 INFO:teuthology.orchestra.run.smithi190.stdout: "index_type": 0 2020-11-09T12:38:51.142 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:38:51.142 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:38:51.142 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:38:51.143 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:38:51.143 INFO:teuthology.orchestra.run.smithi190.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-09T12:38:51.143 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:38:51.144 INFO:teuthology.orchestra.run.smithi190:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-11-09T12:38:51.730 INFO:teuthology.orchestra.run.smithi190.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-11-09T12:38:51.751 INFO:teuthology.orchestra.run.smithi190:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-11-09T12:38:52.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2020-11-09T12:38:52.745 INFO:teuthology.orchestra.run.smithi190.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-11-09T12:38:52.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:52.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:52.765 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T12:38:52.807 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:52.808 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:38:52.829 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:38:52.830 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "c104565e-7c15-4c5e-9b2c-908de1973930", 2020-11-09T12:38:52.830 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "test-zone2", 2020-11-09T12:38:52.830 INFO:teuthology.orchestra.run.smithi190.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-09T12:38:52.830 INFO:teuthology.orchestra.run.smithi190.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-09T12:38:52.830 INFO:teuthology.orchestra.run.smithi190.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-09T12:38:52.831 INFO:teuthology.orchestra.run.smithi190.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-09T12:38:52.831 INFO:teuthology.orchestra.run.smithi190.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-09T12:38:52.831 INFO:teuthology.orchestra.run.smithi190.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-09T12:38:52.831 INFO:teuthology.orchestra.run.smithi190.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-09T12:38:52.831 INFO:teuthology.orchestra.run.smithi190.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-09T12:38:52.831 INFO:teuthology.orchestra.run.smithi190.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-09T12:38:52.831 INFO:teuthology.orchestra.run.smithi190.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-09T12:38:52.832 INFO:teuthology.orchestra.run.smithi190.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-09T12:38:52.832 INFO:teuthology.orchestra.run.smithi190.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-09T12:38:52.832 INFO:teuthology.orchestra.run.smithi190.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-09T12:38:52.832 INFO:teuthology.orchestra.run.smithi190.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-09T12:38:52.832 INFO:teuthology.orchestra.run.smithi190.stdout: "system_key": { 2020-11-09T12:38:52.832 INFO:teuthology.orchestra.run.smithi190.stdout: "access_key": "JX6ROEU3B1OMPZYL", 2020-11-09T12:38:52.832 INFO:teuthology.orchestra.run.smithi190.stdout: "secret_key": "GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme" 2020-11-09T12:38:52.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:38:52.833 INFO:teuthology.orchestra.run.smithi190.stdout: "placement_pools": [ 2020-11-09T12:38:52.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:38:52.833 INFO:teuthology.orchestra.run.smithi190.stdout: "key": "default-placement", 2020-11-09T12:38:52.833 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:38:52.833 INFO:teuthology.orchestra.run.smithi190.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-09T12:38:52.833 INFO:teuthology.orchestra.run.smithi190.stdout: "storage_classes": { 2020-11-09T12:38:52.833 INFO:teuthology.orchestra.run.smithi190.stdout: "STANDARD": { 2020-11-09T12:38:52.834 INFO:teuthology.orchestra.run.smithi190.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-09T12:38:52.834 INFO:teuthology.orchestra.run.smithi190.stdout: "compression_type": "random" 2020-11-09T12:38:52.834 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:38:52.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:38:52.834 INFO:teuthology.orchestra.run.smithi190.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-09T12:38:52.834 INFO:teuthology.orchestra.run.smithi190.stdout: "index_type": 0 2020-11-09T12:38:52.834 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:38:52.835 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:38:52.835 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:38:52.835 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_id": "", 2020-11-09T12:38:52.835 INFO:teuthology.orchestra.run.smithi190.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-09T12:38:52.835 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:38:52.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:52.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:38:52.837 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T12:38:52.918 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:38:52.919 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:39:00.930 INFO:teuthology.orchestra.run.smithi190.stderr:Sending period to new master zone 6dfaac43-4458-47cb-abae-96e6cb987c71 2020-11-09T12:39:01.030 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:39:01.031 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:39:01.031 INFO:teuthology.orchestra.run.smithi190.stdout: "epoch": 2, 2020-11-09T12:39:01.031 INFO:teuthology.orchestra.run.smithi190.stdout: "predecessor_uuid": "411b4533-24c9-4015-aaf1-5ab40f2fe3c0", 2020-11-09T12:39:01.031 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": [], 2020-11-09T12:39:01.031 INFO:teuthology.orchestra.run.smithi190.stdout: "period_map": { 2020-11-09T12:39:01.032 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:39:01.032 INFO:teuthology.orchestra.run.smithi190.stdout: "zonegroups": [ 2020-11-09T12:39:01.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:01.032 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "test-zonegroup", 2020-11-09T12:39:01.032 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "test-zonegroup", 2020-11-09T12:39:01.032 INFO:teuthology.orchestra.run.smithi190.stdout: "api_name": "", 2020-11-09T12:39:01.032 INFO:teuthology.orchestra.run.smithi190.stdout: "is_master": "true", 2020-11-09T12:39:01.033 INFO:teuthology.orchestra.run.smithi190.stdout: "endpoints": [ 2020-11-09T12:39:01.033 INFO:teuthology.orchestra.run.smithi190.stdout: "http://smithi164.front.sepia.ceph.com:8000" 2020-11-09T12:39:01.033 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:01.033 INFO:teuthology.orchestra.run.smithi190.stdout: "hostnames": [], 2020-11-09T12:39:01.033 INFO:teuthology.orchestra.run.smithi190.stdout: "hostnames_s3website": [], 2020-11-09T12:39:01.033 INFO:teuthology.orchestra.run.smithi190.stdout: "master_zone": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:39:01.033 INFO:teuthology.orchestra.run.smithi190.stdout: "zones": [ 2020-11-09T12:39:01.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:01.034 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:39:01.034 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "test-zone1", 2020-11-09T12:39:01.035 INFO:teuthology.orchestra.run.smithi190.stdout: "endpoints": [ 2020-11-09T12:39:01.035 INFO:teuthology.orchestra.run.smithi190.stdout: "http://smithi164.front.sepia.ceph.com:8000" 2020-11-09T12:39:01.035 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:01.035 INFO:teuthology.orchestra.run.smithi190.stdout: "log_meta": "false", 2020-11-09T12:39:01.036 INFO:teuthology.orchestra.run.smithi190.stdout: "log_data": "true", 2020-11-09T12:39:01.036 INFO:teuthology.orchestra.run.smithi190.stdout: "bucket_index_max_shards": 11, 2020-11-09T12:39:01.036 INFO:teuthology.orchestra.run.smithi190.stdout: "read_only": "false", 2020-11-09T12:39:01.036 INFO:teuthology.orchestra.run.smithi190.stdout: "tier_type": "", 2020-11-09T12:39:01.036 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_from_all": "true", 2020-11-09T12:39:01.036 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_from": [], 2020-11-09T12:39:01.036 INFO:teuthology.orchestra.run.smithi190.stdout: "redirect_zone": "" 2020-11-09T12:39:01.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:01.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:01.037 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "c104565e-7c15-4c5e-9b2c-908de1973930", 2020-11-09T12:39:01.037 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "test-zone2", 2020-11-09T12:39:01.037 INFO:teuthology.orchestra.run.smithi190.stdout: "endpoints": [ 2020-11-09T12:39:01.037 INFO:teuthology.orchestra.run.smithi190.stdout: "http://smithi190.front.sepia.ceph.com:8000" 2020-11-09T12:39:01.037 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:01.037 INFO:teuthology.orchestra.run.smithi190.stdout: "log_meta": "false", 2020-11-09T12:39:01.038 INFO:teuthology.orchestra.run.smithi190.stdout: "log_data": "true", 2020-11-09T12:39:01.038 INFO:teuthology.orchestra.run.smithi190.stdout: "bucket_index_max_shards": 11, 2020-11-09T12:39:01.038 INFO:teuthology.orchestra.run.smithi190.stdout: "read_only": "false", 2020-11-09T12:39:01.038 INFO:teuthology.orchestra.run.smithi190.stdout: "tier_type": "", 2020-11-09T12:39:01.038 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_from_all": "true", 2020-11-09T12:39:01.038 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_from": [], 2020-11-09T12:39:01.038 INFO:teuthology.orchestra.run.smithi190.stdout: "redirect_zone": "" 2020-11-09T12:39:01.039 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:01.039 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:01.039 INFO:teuthology.orchestra.run.smithi190.stdout: "placement_targets": [ 2020-11-09T12:39:01.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:01.039 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "default-placement", 2020-11-09T12:39:01.039 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], 2020-11-09T12:39:01.040 INFO:teuthology.orchestra.run.smithi190.stdout: "storage_classes": [ 2020-11-09T12:39:01.040 INFO:teuthology.orchestra.run.smithi190.stdout: "STANDARD" 2020-11-09T12:39:01.040 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:39:01.040 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:01.040 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:01.040 INFO:teuthology.orchestra.run.smithi190.stdout: "default_placement": "default-placement", 2020-11-09T12:39:01.041 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:39:01.041 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_policy": { 2020-11-09T12:39:01.041 INFO:teuthology.orchestra.run.smithi190.stdout: "groups": [] 2020-11-09T12:39:01.041 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:01.042 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:01.042 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:01.042 INFO:teuthology.orchestra.run.smithi190.stdout: "short_zone_ids": [ 2020-11-09T12:39:01.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:01.042 INFO:teuthology.orchestra.run.smithi190.stdout: "key": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:39:01.043 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 1014254849 2020-11-09T12:39:01.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:01.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:01.043 INFO:teuthology.orchestra.run.smithi190.stdout: "key": "c104565e-7c15-4c5e-9b2c-908de1973930", 2020-11-09T12:39:01.043 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 623289585 2020-11-09T12:39:01.044 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:01.044 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:39:01.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:01.044 INFO:teuthology.orchestra.run.smithi190.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T12:39:01.044 INFO:teuthology.orchestra.run.smithi190.stdout: "master_zone": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:39:01.044 INFO:teuthology.orchestra.run.smithi190.stdout: "period_config": { 2020-11-09T12:39:01.045 INFO:teuthology.orchestra.run.smithi190.stdout: "bucket_quota": { 2020-11-09T12:39:01.045 INFO:teuthology.orchestra.run.smithi190.stdout: "enabled": false, 2020-11-09T12:39:01.045 INFO:teuthology.orchestra.run.smithi190.stdout: "check_on_raw": false, 2020-11-09T12:39:01.046 INFO:teuthology.orchestra.run.smithi190.stdout: "max_size": -1, 2020-11-09T12:39:01.046 INFO:teuthology.orchestra.run.smithi190.stdout: "max_size_kb": 0, 2020-11-09T12:39:01.046 INFO:teuthology.orchestra.run.smithi190.stdout: "max_objects": -1 2020-11-09T12:39:01.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:01.047 INFO:teuthology.orchestra.run.smithi190.stdout: "user_quota": { 2020-11-09T12:39:01.047 INFO:teuthology.orchestra.run.smithi190.stdout: "enabled": false, 2020-11-09T12:39:01.047 INFO:teuthology.orchestra.run.smithi190.stdout: "check_on_raw": false, 2020-11-09T12:39:01.047 INFO:teuthology.orchestra.run.smithi190.stdout: "max_size": -1, 2020-11-09T12:39:01.047 INFO:teuthology.orchestra.run.smithi190.stdout: "max_size_kb": 0, 2020-11-09T12:39:01.048 INFO:teuthology.orchestra.run.smithi190.stdout: "max_objects": -1 2020-11-09T12:39:01.048 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:01.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:01.048 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:39:01.048 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_name": "test-realm", 2020-11-09T12:39:01.049 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:01.049 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:39:01.050 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-11-09T12:39:01.050 INFO:teuthology.orchestra.run:waiting for 300 2020-11-09T12:39:01.092 INFO:tasks.rgw.c2.client.0.smithi190.stdout:2020-11-09T12:39:01.090+0000 7f84db7fe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 14538) UID: 0 2020-11-09T12:39:01.093 INFO:tasks.rgw.c2.client.0.smithi190.stdout:2020-11-09T12:39:01.090+0000 7f84ff4a0100 -1 shutting down 2020-11-09T12:39:04.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2020-11-09T12:39:07.152 INFO:tasks.rgw.c2.client.0:Stopped 2020-11-09T12:39:07.153 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-11-09T12:39:07.153 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-11-09T12:39:07.156 INFO:tasks.rgw.c2.client.0:Started 2020-11-09T12:39:07.156 INFO:teuthology.orchestra.run.smithi190:> curl http://smithi190.front.sepia.ceph.com:8000 2020-11-09T12:39:07.173 INFO:teuthology.orchestra.run.smithi190.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T12:39:07.174 INFO:teuthology.orchestra.run.smithi190.stderr: Dload Upload Total Spent Left Speed 2020-11-09T12:39:07.177 INFO:teuthology.orchestra.run.smithi190.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi190.front.sepia.ceph.com port 8000: Connection refused 2020-11-09T12:39:07.178 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-09T12:39:08.179 INFO:teuthology.orchestra.run.smithi190:> curl http://smithi190.front.sepia.ceph.com:8000 2020-11-09T12:39:08.196 INFO:teuthology.orchestra.run.smithi190.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T12:39:08.197 INFO:teuthology.orchestra.run.smithi190.stderr: Dload Upload Total Spent Left Speed 2020-11-09T12:39:08.202 INFO:teuthology.orchestra.run.smithi190.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-09T12:39:08.203 INFO:teuthology.orchestra.run.smithi190.stdout:anonymous 2020-11-09T12:39:08.204 INFO:tasks.rgw_multisite:creating zone.. 2020-11-09T12:39:08.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--endpoints', 'http://smithi164.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:08.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--endpoints', 'http://smithi164.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:08.205 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key JX6ROEU3B1OMPZYL --secret GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme --endpoints http://smithi164.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T12:39:08.248 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:39:08.249 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:39:08.313 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:39:08.311+0000 7f24488740c0 0 failed reading obj info from .rgw.root:zone_info.c104565e-7c15-4c5e-9b2c-908de1973930: (2) No such file or directory 2020-11-09T12:39:08.313 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:39:08.311+0000 7f24488740c0 0 WARNING: could not read zone params for zone id=c104565e-7c15-4c5e-9b2c-908de1973930 name=test-zone2 2020-11-09T12:39:08.321 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:39:08.322 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "bd4c3e31-4a1f-445d-9b97-3f53c3f0f6b3", 2020-11-09T12:39:08.322 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "test-zone3", 2020-11-09T12:39:08.322 INFO:teuthology.orchestra.run.smithi164.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-11-09T12:39:08.322 INFO:teuthology.orchestra.run.smithi164.stdout: "control_pool": "test-zone3.rgw.control", 2020-11-09T12:39:08.322 INFO:teuthology.orchestra.run.smithi164.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-11-09T12:39:08.323 INFO:teuthology.orchestra.run.smithi164.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-11-09T12:39:08.323 INFO:teuthology.orchestra.run.smithi164.stdout: "log_pool": "test-zone3.rgw.log", 2020-11-09T12:39:08.323 INFO:teuthology.orchestra.run.smithi164.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-11-09T12:39:08.323 INFO:teuthology.orchestra.run.smithi164.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-11-09T12:39:08.323 INFO:teuthology.orchestra.run.smithi164.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-11-09T12:39:08.323 INFO:teuthology.orchestra.run.smithi164.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-11-09T12:39:08.324 INFO:teuthology.orchestra.run.smithi164.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-11-09T12:39:08.324 INFO:teuthology.orchestra.run.smithi164.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-11-09T12:39:08.324 INFO:teuthology.orchestra.run.smithi164.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-11-09T12:39:08.324 INFO:teuthology.orchestra.run.smithi164.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-11-09T12:39:08.325 INFO:teuthology.orchestra.run.smithi164.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-11-09T12:39:08.325 INFO:teuthology.orchestra.run.smithi164.stdout: "system_key": { 2020-11-09T12:39:08.325 INFO:teuthology.orchestra.run.smithi164.stdout: "access_key": "JX6ROEU3B1OMPZYL", 2020-11-09T12:39:08.325 INFO:teuthology.orchestra.run.smithi164.stdout: "secret_key": "GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme" 2020-11-09T12:39:08.325 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:08.325 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_pools": [ 2020-11-09T12:39:08.326 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:08.326 INFO:teuthology.orchestra.run.smithi164.stdout: "key": "default-placement", 2020-11-09T12:39:08.326 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:39:08.326 INFO:teuthology.orchestra.run.smithi164.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-11-09T12:39:08.326 INFO:teuthology.orchestra.run.smithi164.stdout: "storage_classes": { 2020-11-09T12:39:08.326 INFO:teuthology.orchestra.run.smithi164.stdout: "STANDARD": { 2020-11-09T12:39:08.327 INFO:teuthology.orchestra.run.smithi164.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2020-11-09T12:39:08.327 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:39:08.327 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:08.327 INFO:teuthology.orchestra.run.smithi164.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-11-09T12:39:08.327 INFO:teuthology.orchestra.run.smithi164.stdout: "index_type": 0 2020-11-09T12:39:08.327 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:39:08.327 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:39:08.327 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:39:08.328 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:39:08.328 INFO:teuthology.orchestra.run.smithi164.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-11-09T12:39:08.328 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:39:08.329 INFO:teuthology.orchestra.run.smithi164:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-11-09T12:39:09.100 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-11-09T12:39:09.125 INFO:teuthology.orchestra.run.smithi164:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-11-09T12:39:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-11-09T12:39:10.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:10.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:10.117 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T12:39:10.164 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:39:10.164 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:39:10.191 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:39:10.191 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "bd4c3e31-4a1f-445d-9b97-3f53c3f0f6b3", 2020-11-09T12:39:10.191 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "test-zone3", 2020-11-09T12:39:10.191 INFO:teuthology.orchestra.run.smithi164.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-11-09T12:39:10.192 INFO:teuthology.orchestra.run.smithi164.stdout: "control_pool": "test-zone3.rgw.control", 2020-11-09T12:39:10.192 INFO:teuthology.orchestra.run.smithi164.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-11-09T12:39:10.192 INFO:teuthology.orchestra.run.smithi164.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-11-09T12:39:10.192 INFO:teuthology.orchestra.run.smithi164.stdout: "log_pool": "test-zone3.rgw.log", 2020-11-09T12:39:10.192 INFO:teuthology.orchestra.run.smithi164.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-11-09T12:39:10.192 INFO:teuthology.orchestra.run.smithi164.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-11-09T12:39:10.193 INFO:teuthology.orchestra.run.smithi164.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-11-09T12:39:10.193 INFO:teuthology.orchestra.run.smithi164.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-11-09T12:39:10.193 INFO:teuthology.orchestra.run.smithi164.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-11-09T12:39:10.193 INFO:teuthology.orchestra.run.smithi164.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-11-09T12:39:10.193 INFO:teuthology.orchestra.run.smithi164.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-11-09T12:39:10.193 INFO:teuthology.orchestra.run.smithi164.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-11-09T12:39:10.193 INFO:teuthology.orchestra.run.smithi164.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-11-09T12:39:10.193 INFO:teuthology.orchestra.run.smithi164.stdout: "system_key": { 2020-11-09T12:39:10.194 INFO:teuthology.orchestra.run.smithi164.stdout: "access_key": "JX6ROEU3B1OMPZYL", 2020-11-09T12:39:10.194 INFO:teuthology.orchestra.run.smithi164.stdout: "secret_key": "GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme" 2020-11-09T12:39:10.194 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:10.194 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_pools": [ 2020-11-09T12:39:10.194 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:10.194 INFO:teuthology.orchestra.run.smithi164.stdout: "key": "default-placement", 2020-11-09T12:39:10.194 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:39:10.195 INFO:teuthology.orchestra.run.smithi164.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-11-09T12:39:10.195 INFO:teuthology.orchestra.run.smithi164.stdout: "storage_classes": { 2020-11-09T12:39:10.195 INFO:teuthology.orchestra.run.smithi164.stdout: "STANDARD": { 2020-11-09T12:39:10.195 INFO:teuthology.orchestra.run.smithi164.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2020-11-09T12:39:10.195 INFO:teuthology.orchestra.run.smithi164.stdout: "compression_type": "random" 2020-11-09T12:39:10.195 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:39:10.195 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:10.196 INFO:teuthology.orchestra.run.smithi164.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-11-09T12:39:10.196 INFO:teuthology.orchestra.run.smithi164.stdout: "index_type": 0 2020-11-09T12:39:10.196 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:39:10.196 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:39:10.196 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:39:10.196 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "", 2020-11-09T12:39:10.196 INFO:teuthology.orchestra.run.smithi164.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-11-09T12:39:10.197 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:39:10.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:10.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:10.198 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T12:39:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:39:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:39:10.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2020-11-09T12:39:18.328 INFO:teuthology.orchestra.run.smithi164.stderr:Sending period to new master zone 6dfaac43-4458-47cb-abae-96e6cb987c71 2020-11-09T12:39:18.400 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:39:18.401 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:39:18.401 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 3, 2020-11-09T12:39:18.401 INFO:teuthology.orchestra.run.smithi164.stdout: "predecessor_uuid": "411b4533-24c9-4015-aaf1-5ab40f2fe3c0", 2020-11-09T12:39:18.402 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": [], 2020-11-09T12:39:18.402 INFO:teuthology.orchestra.run.smithi164.stdout: "period_map": { 2020-11-09T12:39:18.402 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:39:18.404 INFO:teuthology.orchestra.run.smithi164.stdout: "zonegroups": [ 2020-11-09T12:39:18.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:18.404 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "test-zonegroup", 2020-11-09T12:39:18.404 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "test-zonegroup", 2020-11-09T12:39:18.405 INFO:teuthology.orchestra.run.smithi164.stdout: "api_name": "", 2020-11-09T12:39:18.405 INFO:teuthology.orchestra.run.smithi164.stdout: "is_master": "true", 2020-11-09T12:39:18.405 INFO:teuthology.orchestra.run.smithi164.stdout: "endpoints": [ 2020-11-09T12:39:18.405 INFO:teuthology.orchestra.run.smithi164.stdout: "http://smithi164.front.sepia.ceph.com:8000" 2020-11-09T12:39:18.405 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:39:18.407 INFO:teuthology.orchestra.run.smithi164.stdout: "hostnames": [], 2020-11-09T12:39:18.407 INFO:teuthology.orchestra.run.smithi164.stdout: "hostnames_s3website": [], 2020-11-09T12:39:18.407 INFO:teuthology.orchestra.run.smithi164.stdout: "master_zone": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:39:18.407 INFO:teuthology.orchestra.run.smithi164.stdout: "zones": [ 2020-11-09T12:39:18.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:18.408 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:39:18.408 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "test-zone1", 2020-11-09T12:39:18.408 INFO:teuthology.orchestra.run.smithi164.stdout: "endpoints": [ 2020-11-09T12:39:18.408 INFO:teuthology.orchestra.run.smithi164.stdout: "http://smithi164.front.sepia.ceph.com:8000" 2020-11-09T12:39:18.409 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:39:18.410 INFO:teuthology.orchestra.run.smithi164.stdout: "log_meta": "false", 2020-11-09T12:39:18.410 INFO:teuthology.orchestra.run.smithi164.stdout: "log_data": "true", 2020-11-09T12:39:18.411 INFO:teuthology.orchestra.run.smithi164.stdout: "bucket_index_max_shards": 11, 2020-11-09T12:39:18.411 INFO:teuthology.orchestra.run.smithi164.stdout: "read_only": "false", 2020-11-09T12:39:18.411 INFO:teuthology.orchestra.run.smithi164.stdout: "tier_type": "", 2020-11-09T12:39:18.411 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_from_all": "true", 2020-11-09T12:39:18.411 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_from": [], 2020-11-09T12:39:18.411 INFO:teuthology.orchestra.run.smithi164.stdout: "redirect_zone": "" 2020-11-09T12:39:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "bd4c3e31-4a1f-445d-9b97-3f53c3f0f6b3", 2020-11-09T12:39:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "test-zone3", 2020-11-09T12:39:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: "endpoints": [ 2020-11-09T12:39:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: "http://smithi164.front.sepia.ceph.com:8001" 2020-11-09T12:39:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:39:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "log_meta": "false", 2020-11-09T12:39:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "log_data": "true", 2020-11-09T12:39:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "bucket_index_max_shards": 11, 2020-11-09T12:39:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "read_only": "false", 2020-11-09T12:39:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "tier_type": "", 2020-11-09T12:39:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_from_all": "true", 2020-11-09T12:39:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_from": [], 2020-11-09T12:39:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "redirect_zone": "" 2020-11-09T12:39:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "c104565e-7c15-4c5e-9b2c-908de1973930", 2020-11-09T12:39:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "test-zone2", 2020-11-09T12:39:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "endpoints": [ 2020-11-09T12:39:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "http://smithi190.front.sepia.ceph.com:8000" 2020-11-09T12:39:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:39:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "log_meta": "false", 2020-11-09T12:39:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "log_data": "true", 2020-11-09T12:39:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "bucket_index_max_shards": 11, 2020-11-09T12:39:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "read_only": "false", 2020-11-09T12:39:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "tier_type": "", 2020-11-09T12:39:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_from_all": "true", 2020-11-09T12:39:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_from": [], 2020-11-09T12:39:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "redirect_zone": "" 2020-11-09T12:39:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:39:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:39:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_targets": [ 2020-11-09T12:39:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "default-placement", 2020-11-09T12:39:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: "tags": [], 2020-11-09T12:39:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: "storage_classes": [ 2020-11-09T12:39:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: "STANDARD" 2020-11-09T12:39:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:39:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:39:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:39:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: "default_placement": "default-placement", 2020-11-09T12:39:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:39:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_policy": { 2020-11-09T12:39:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: "groups": [] 2020-11-09T12:39:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:39:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:39:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:39:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: "short_zone_ids": [ 2020-11-09T12:39:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: "key": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:39:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 1014254849 2020-11-09T12:39:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: "key": "bd4c3e31-4a1f-445d-9b97-3f53c3f0f6b3", 2020-11-09T12:39:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 2419824616 2020-11-09T12:39:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: "key": "c104565e-7c15-4c5e-9b2c-908de1973930", 2020-11-09T12:39:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 623289585 2020-11-09T12:39:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:39:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:39:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T12:39:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: "master_zone": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:39:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: "period_config": { 2020-11-09T12:39:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: "bucket_quota": { 2020-11-09T12:39:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: "enabled": false, 2020-11-09T12:39:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: "check_on_raw": false, 2020-11-09T12:39:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size": -1, 2020-11-09T12:39:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size_kb": 0, 2020-11-09T12:39:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: "max_objects": -1 2020-11-09T12:39:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:18.424 INFO:teuthology.orchestra.run.smithi164.stdout: "user_quota": { 2020-11-09T12:39:18.424 INFO:teuthology.orchestra.run.smithi164.stdout: "enabled": false, 2020-11-09T12:39:18.425 INFO:teuthology.orchestra.run.smithi164.stdout: "check_on_raw": false, 2020-11-09T12:39:18.428 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size": -1, 2020-11-09T12:39:18.428 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size_kb": 0, 2020-11-09T12:39:18.428 INFO:teuthology.orchestra.run.smithi164.stdout: "max_objects": -1 2020-11-09T12:39:18.429 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:39:18.429 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:18.429 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:39:18.429 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_name": "test-realm", 2020-11-09T12:39:18.429 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:39:18.429 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:39:18.431 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-11-09T12:39:18.431 INFO:teuthology.orchestra.run:waiting for 300 2020-11-09T12:39:18.477 INFO:tasks.rgw.c1.client.1.smithi164.stdout:2020-11-09T12:39:18.475+0000 7f6fa7fff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 14497) UID: 0 2020-11-09T12:39:18.478 INFO:tasks.rgw.c1.client.1.smithi164.stdout:2020-11-09T12:39:18.475+0000 7f6fcbeae100 -1 shutting down 2020-11-09T12:39:20.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-11-09T12:39:24.533 INFO:tasks.rgw.c1.client.1:Stopped 2020-11-09T12:39:24.533 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-09T12:39:24.534 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-09T12:39:24.537 INFO:tasks.rgw.c1.client.1:Started 2020-11-09T12:39:24.537 INFO:teuthology.orchestra.run.smithi164:> curl http://smithi164.front.sepia.ceph.com:8001 2020-11-09T12:39:24.554 INFO:teuthology.orchestra.run.smithi164.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T12:39:24.554 INFO:teuthology.orchestra.run.smithi164.stderr: Dload Upload Total Spent Left Speed 2020-11-09T12:39:24.558 INFO:teuthology.orchestra.run.smithi164.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi164.front.sepia.ceph.com port 8001: Connection refused 2020-11-09T12:39:24.558 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-09T12:39:25.560 INFO:teuthology.orchestra.run.smithi164:> curl http://smithi164.front.sepia.ceph.com:8001 2020-11-09T12:39:25.576 INFO:teuthology.orchestra.run.smithi164.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T12:39:25.576 INFO:teuthology.orchestra.run.smithi164.stderr: Dload Upload Total Spent Left Speed 2020-11-09T12:39:25.582 INFO:teuthology.orchestra.run.smithi164.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2020-11-09T12:39:25.583 INFO:teuthology.orchestra.run.smithi164.stdout:anonymous 2020-11-09T12:39:25.584 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-11-09T12:39:25.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi164.front.sepia.ceph.com:8000', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--url', 'http://smithi164.front.sepia.ceph.com:8000', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:25.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi164.front.sepia.ceph.com:8000', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--url', 'http://smithi164.front.sepia.ceph.com:8000', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:25.585 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi164.front.sepia.ceph.com:8000 --access-key JX6ROEU3B1OMPZYL --secret GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme --url http://smithi164.front.sepia.ceph.com:8000 --access-key JX6ROEU3B1OMPZYL --secret GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T12:39:25.632 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:39:25.633 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:39:25.670 INFO:tasks.rgw.c2.client.0.smithi190.stdout:2020-11-09T12:39:25.666+0000 7fd4f6614700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.32656115-d19e-4547-bae3-a6fce59494ce.control err=-107 2020-11-09T12:39:25.699 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:39:25.700 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:39:25.700 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "test-realm", 2020-11-09T12:39:25.700 INFO:teuthology.orchestra.run.smithi190.stdout: "current_period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:39:25.700 INFO:teuthology.orchestra.run.smithi190.stdout: "epoch": 2 2020-11-09T12:39:25.701 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:39:25.702 INFO:tasks.rgw_multisite:creating zone.. 2020-11-09T12:39:25.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--endpoints', 'http://smithi190.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:25.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'JX6ROEU3B1OMPZYL', '--secret', 'GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme', '--endpoints', 'http://smithi190.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:25.703 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key JX6ROEU3B1OMPZYL --secret GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme --endpoints http://smithi190.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T12:39:25.754 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:39:25.754 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:39:25.800 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:39:25.798+0000 7f8984d910c0 0 failed reading obj info from .rgw.root:zone_info.6dfaac43-4458-47cb-abae-96e6cb987c71: (2) No such file or directory 2020-11-09T12:39:25.800 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:39:25.798+0000 7f8984d910c0 0 WARNING: could not read zone params for zone id=6dfaac43-4458-47cb-abae-96e6cb987c71 name=test-zone1 2020-11-09T12:39:25.800 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:39:25.798+0000 7f8984d910c0 0 failed reading obj info from .rgw.root:zone_info.bd4c3e31-4a1f-445d-9b97-3f53c3f0f6b3: (2) No such file or directory 2020-11-09T12:39:25.801 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:39:25.798+0000 7f8984d910c0 0 WARNING: could not read zone params for zone id=bd4c3e31-4a1f-445d-9b97-3f53c3f0f6b3 name=test-zone3 2020-11-09T12:39:25.808 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:39:25.809 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "e563eedc-cfbd-426b-a4d4-572da58c3b13", 2020-11-09T12:39:25.809 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "test-zone4", 2020-11-09T12:39:25.809 INFO:teuthology.orchestra.run.smithi190.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-11-09T12:39:25.809 INFO:teuthology.orchestra.run.smithi190.stdout: "control_pool": "test-zone4.rgw.control", 2020-11-09T12:39:25.809 INFO:teuthology.orchestra.run.smithi190.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-11-09T12:39:25.809 INFO:teuthology.orchestra.run.smithi190.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-11-09T12:39:25.809 INFO:teuthology.orchestra.run.smithi190.stdout: "log_pool": "test-zone4.rgw.log", 2020-11-09T12:39:25.810 INFO:teuthology.orchestra.run.smithi190.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-11-09T12:39:25.810 INFO:teuthology.orchestra.run.smithi190.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-11-09T12:39:25.810 INFO:teuthology.orchestra.run.smithi190.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-11-09T12:39:25.810 INFO:teuthology.orchestra.run.smithi190.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-11-09T12:39:25.810 INFO:teuthology.orchestra.run.smithi190.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-11-09T12:39:25.810 INFO:teuthology.orchestra.run.smithi190.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-11-09T12:39:25.811 INFO:teuthology.orchestra.run.smithi190.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-11-09T12:39:25.811 INFO:teuthology.orchestra.run.smithi190.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-11-09T12:39:25.811 INFO:teuthology.orchestra.run.smithi190.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-11-09T12:39:25.811 INFO:teuthology.orchestra.run.smithi190.stdout: "system_key": { 2020-11-09T12:39:25.811 INFO:teuthology.orchestra.run.smithi190.stdout: "access_key": "JX6ROEU3B1OMPZYL", 2020-11-09T12:39:25.811 INFO:teuthology.orchestra.run.smithi190.stdout: "secret_key": "GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme" 2020-11-09T12:39:25.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:25.812 INFO:teuthology.orchestra.run.smithi190.stdout: "placement_pools": [ 2020-11-09T12:39:25.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:25.812 INFO:teuthology.orchestra.run.smithi190.stdout: "key": "default-placement", 2020-11-09T12:39:25.812 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:39:25.812 INFO:teuthology.orchestra.run.smithi190.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-11-09T12:39:25.812 INFO:teuthology.orchestra.run.smithi190.stdout: "storage_classes": { 2020-11-09T12:39:25.812 INFO:teuthology.orchestra.run.smithi190.stdout: "STANDARD": { 2020-11-09T12:39:25.813 INFO:teuthology.orchestra.run.smithi190.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2020-11-09T12:39:25.813 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:25.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:25.813 INFO:teuthology.orchestra.run.smithi190.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-11-09T12:39:25.813 INFO:teuthology.orchestra.run.smithi190.stdout: "index_type": 0 2020-11-09T12:39:25.813 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:25.813 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:25.814 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:25.814 INFO:teuthology.orchestra.run.smithi190.stdout: "tier_config": { 2020-11-09T12:39:25.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event_retention_days": 7, 2020-11-09T12:39:25.814 INFO:teuthology.orchestra.run.smithi190.stdout: "start_with_full_sync": false 2020-11-09T12:39:25.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:25.814 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:39:25.814 INFO:teuthology.orchestra.run.smithi190.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-11-09T12:39:25.815 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:39:25.816 INFO:teuthology.orchestra.run.smithi190:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-11-09T12:39:26.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-11-09T12:39:26.703 INFO:teuthology.orchestra.run.smithi190.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-11-09T12:39:26.733 INFO:teuthology.orchestra.run.smithi190:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-11-09T12:39:27.732 INFO:teuthology.orchestra.run.smithi190.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-11-09T12:39:27.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:27.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:27.760 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T12:39:27.807 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:39:27.807 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:39:27.828 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:39:27.828 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "e563eedc-cfbd-426b-a4d4-572da58c3b13", 2020-11-09T12:39:27.828 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "test-zone4", 2020-11-09T12:39:27.828 INFO:teuthology.orchestra.run.smithi190.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-11-09T12:39:27.829 INFO:teuthology.orchestra.run.smithi190.stdout: "control_pool": "test-zone4.rgw.control", 2020-11-09T12:39:27.829 INFO:teuthology.orchestra.run.smithi190.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-11-09T12:39:27.829 INFO:teuthology.orchestra.run.smithi190.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-11-09T12:39:27.829 INFO:teuthology.orchestra.run.smithi190.stdout: "log_pool": "test-zone4.rgw.log", 2020-11-09T12:39:27.829 INFO:teuthology.orchestra.run.smithi190.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-11-09T12:39:27.829 INFO:teuthology.orchestra.run.smithi190.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-11-09T12:39:27.829 INFO:teuthology.orchestra.run.smithi190.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-11-09T12:39:27.830 INFO:teuthology.orchestra.run.smithi190.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-11-09T12:39:27.830 INFO:teuthology.orchestra.run.smithi190.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-11-09T12:39:27.830 INFO:teuthology.orchestra.run.smithi190.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-11-09T12:39:27.830 INFO:teuthology.orchestra.run.smithi190.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-11-09T12:39:27.830 INFO:teuthology.orchestra.run.smithi190.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-11-09T12:39:27.830 INFO:teuthology.orchestra.run.smithi190.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-11-09T12:39:27.831 INFO:teuthology.orchestra.run.smithi190.stdout: "system_key": { 2020-11-09T12:39:27.831 INFO:teuthology.orchestra.run.smithi190.stdout: "access_key": "JX6ROEU3B1OMPZYL", 2020-11-09T12:39:27.831 INFO:teuthology.orchestra.run.smithi190.stdout: "secret_key": "GqDhlPLnCZZeMpna1Fi1iMakOxcxjNme" 2020-11-09T12:39:27.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:27.832 INFO:teuthology.orchestra.run.smithi190.stdout: "placement_pools": [ 2020-11-09T12:39:27.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:27.832 INFO:teuthology.orchestra.run.smithi190.stdout: "key": "default-placement", 2020-11-09T12:39:27.832 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:39:27.832 INFO:teuthology.orchestra.run.smithi190.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-11-09T12:39:27.832 INFO:teuthology.orchestra.run.smithi190.stdout: "storage_classes": { 2020-11-09T12:39:27.832 INFO:teuthology.orchestra.run.smithi190.stdout: "STANDARD": { 2020-11-09T12:39:27.833 INFO:teuthology.orchestra.run.smithi190.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2020-11-09T12:39:27.833 INFO:teuthology.orchestra.run.smithi190.stdout: "compression_type": "random" 2020-11-09T12:39:27.833 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:27.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:27.833 INFO:teuthology.orchestra.run.smithi190.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-11-09T12:39:27.833 INFO:teuthology.orchestra.run.smithi190.stdout: "index_type": 0 2020-11-09T12:39:27.833 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:27.834 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:27.834 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:27.834 INFO:teuthology.orchestra.run.smithi190.stdout: "tier_config": { 2020-11-09T12:39:27.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event_retention_days": 7, 2020-11-09T12:39:27.834 INFO:teuthology.orchestra.run.smithi190.stdout: "start_with_full_sync": false 2020-11-09T12:39:27.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:27.834 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_id": "", 2020-11-09T12:39:27.835 INFO:teuthology.orchestra.run.smithi190.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-11-09T12:39:27.835 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:39:27.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:27.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:27.836 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-09T12:39:27.874 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:39:27.875 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:39:35.914 INFO:teuthology.orchestra.run.smithi190.stderr:Sending period to new master zone 6dfaac43-4458-47cb-abae-96e6cb987c71 2020-11-09T12:39:35.983 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:39:35.983 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:39:35.984 INFO:teuthology.orchestra.run.smithi190.stdout: "epoch": 4, 2020-11-09T12:39:35.984 INFO:teuthology.orchestra.run.smithi190.stdout: "predecessor_uuid": "411b4533-24c9-4015-aaf1-5ab40f2fe3c0", 2020-11-09T12:39:35.984 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": [], 2020-11-09T12:39:35.984 INFO:teuthology.orchestra.run.smithi190.stdout: "period_map": { 2020-11-09T12:39:35.985 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:39:35.985 INFO:teuthology.orchestra.run.smithi190.stdout: "zonegroups": [ 2020-11-09T12:39:35.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:35.985 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "test-zonegroup", 2020-11-09T12:39:35.986 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "test-zonegroup", 2020-11-09T12:39:35.986 INFO:teuthology.orchestra.run.smithi190.stdout: "api_name": "", 2020-11-09T12:39:35.986 INFO:teuthology.orchestra.run.smithi190.stdout: "is_master": "true", 2020-11-09T12:39:35.986 INFO:teuthology.orchestra.run.smithi190.stdout: "endpoints": [ 2020-11-09T12:39:35.986 INFO:teuthology.orchestra.run.smithi190.stdout: "http://smithi164.front.sepia.ceph.com:8000" 2020-11-09T12:39:35.987 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:35.987 INFO:teuthology.orchestra.run.smithi190.stdout: "hostnames": [], 2020-11-09T12:39:35.987 INFO:teuthology.orchestra.run.smithi190.stdout: "hostnames_s3website": [], 2020-11-09T12:39:35.987 INFO:teuthology.orchestra.run.smithi190.stdout: "master_zone": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:39:35.987 INFO:teuthology.orchestra.run.smithi190.stdout: "zones": [ 2020-11-09T12:39:35.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:35.987 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:39:35.988 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "test-zone1", 2020-11-09T12:39:35.988 INFO:teuthology.orchestra.run.smithi190.stdout: "endpoints": [ 2020-11-09T12:39:35.988 INFO:teuthology.orchestra.run.smithi190.stdout: "http://smithi164.front.sepia.ceph.com:8000" 2020-11-09T12:39:35.988 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:35.988 INFO:teuthology.orchestra.run.smithi190.stdout: "log_meta": "false", 2020-11-09T12:39:35.988 INFO:teuthology.orchestra.run.smithi190.stdout: "log_data": "true", 2020-11-09T12:39:35.988 INFO:teuthology.orchestra.run.smithi190.stdout: "bucket_index_max_shards": 11, 2020-11-09T12:39:35.989 INFO:teuthology.orchestra.run.smithi190.stdout: "read_only": "false", 2020-11-09T12:39:35.989 INFO:teuthology.orchestra.run.smithi190.stdout: "tier_type": "", 2020-11-09T12:39:35.989 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_from_all": "true", 2020-11-09T12:39:35.989 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_from": [], 2020-11-09T12:39:35.989 INFO:teuthology.orchestra.run.smithi190.stdout: "redirect_zone": "" 2020-11-09T12:39:35.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:35.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:35.989 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "bd4c3e31-4a1f-445d-9b97-3f53c3f0f6b3", 2020-11-09T12:39:35.990 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "test-zone3", 2020-11-09T12:39:35.990 INFO:teuthology.orchestra.run.smithi190.stdout: "endpoints": [ 2020-11-09T12:39:35.990 INFO:teuthology.orchestra.run.smithi190.stdout: "http://smithi164.front.sepia.ceph.com:8001" 2020-11-09T12:39:35.990 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:35.990 INFO:teuthology.orchestra.run.smithi190.stdout: "log_meta": "false", 2020-11-09T12:39:35.991 INFO:teuthology.orchestra.run.smithi190.stdout: "log_data": "true", 2020-11-09T12:39:35.991 INFO:teuthology.orchestra.run.smithi190.stdout: "bucket_index_max_shards": 11, 2020-11-09T12:39:35.991 INFO:teuthology.orchestra.run.smithi190.stdout: "read_only": "false", 2020-11-09T12:39:35.991 INFO:teuthology.orchestra.run.smithi190.stdout: "tier_type": "", 2020-11-09T12:39:35.991 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_from_all": "true", 2020-11-09T12:39:35.991 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_from": [], 2020-11-09T12:39:35.991 INFO:teuthology.orchestra.run.smithi190.stdout: "redirect_zone": "" 2020-11-09T12:39:35.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:35.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:35.992 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "c104565e-7c15-4c5e-9b2c-908de1973930", 2020-11-09T12:39:35.992 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "test-zone2", 2020-11-09T12:39:35.992 INFO:teuthology.orchestra.run.smithi190.stdout: "endpoints": [ 2020-11-09T12:39:35.992 INFO:teuthology.orchestra.run.smithi190.stdout: "http://smithi190.front.sepia.ceph.com:8000" 2020-11-09T12:39:35.992 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:35.993 INFO:teuthology.orchestra.run.smithi190.stdout: "log_meta": "false", 2020-11-09T12:39:35.993 INFO:teuthology.orchestra.run.smithi190.stdout: "log_data": "true", 2020-11-09T12:39:35.993 INFO:teuthology.orchestra.run.smithi190.stdout: "bucket_index_max_shards": 11, 2020-11-09T12:39:35.993 INFO:teuthology.orchestra.run.smithi190.stdout: "read_only": "false", 2020-11-09T12:39:35.993 INFO:teuthology.orchestra.run.smithi190.stdout: "tier_type": "", 2020-11-09T12:39:35.994 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_from_all": "true", 2020-11-09T12:39:35.994 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_from": [], 2020-11-09T12:39:35.994 INFO:teuthology.orchestra.run.smithi190.stdout: "redirect_zone": "" 2020-11-09T12:39:35.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:35.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:35.995 INFO:teuthology.orchestra.run.smithi190.stdout: "id": "e563eedc-cfbd-426b-a4d4-572da58c3b13", 2020-11-09T12:39:35.995 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "test-zone4", 2020-11-09T12:39:35.995 INFO:teuthology.orchestra.run.smithi190.stdout: "endpoints": [ 2020-11-09T12:39:35.996 INFO:teuthology.orchestra.run.smithi190.stdout: "http://smithi190.front.sepia.ceph.com:8001" 2020-11-09T12:39:35.996 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:35.996 INFO:teuthology.orchestra.run.smithi190.stdout: "log_meta": "false", 2020-11-09T12:39:35.997 INFO:teuthology.orchestra.run.smithi190.stdout: "log_data": "true", 2020-11-09T12:39:35.997 INFO:teuthology.orchestra.run.smithi190.stdout: "bucket_index_max_shards": 11, 2020-11-09T12:39:35.997 INFO:teuthology.orchestra.run.smithi190.stdout: "read_only": "false", 2020-11-09T12:39:35.997 INFO:teuthology.orchestra.run.smithi190.stdout: "tier_type": "pubsub", 2020-11-09T12:39:35.998 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_from_all": "false", 2020-11-09T12:39:35.998 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_from": [ 2020-11-09T12:39:35.998 INFO:teuthology.orchestra.run.smithi190.stdout: "test-zone1" 2020-11-09T12:39:35.998 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:35.998 INFO:teuthology.orchestra.run.smithi190.stdout: "redirect_zone": "" 2020-11-09T12:39:35.999 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:35.999 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:35.999 INFO:teuthology.orchestra.run.smithi190.stdout: "placement_targets": [ 2020-11-09T12:39:35.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:35.999 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "default-placement", 2020-11-09T12:39:36.000 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], 2020-11-09T12:39:36.000 INFO:teuthology.orchestra.run.smithi190.stdout: "storage_classes": [ 2020-11-09T12:39:36.000 INFO:teuthology.orchestra.run.smithi190.stdout: "STANDARD" 2020-11-09T12:39:36.000 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:39:36.000 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:36.001 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:36.001 INFO:teuthology.orchestra.run.smithi190.stdout: "default_placement": "default-placement", 2020-11-09T12:39:36.001 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:39:36.001 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_policy": { 2020-11-09T12:39:36.001 INFO:teuthology.orchestra.run.smithi190.stdout: "groups": [] 2020-11-09T12:39:36.001 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:36.001 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:36.002 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2020-11-09T12:39:36.002 INFO:teuthology.orchestra.run.smithi190.stdout: "short_zone_ids": [ 2020-11-09T12:39:36.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:36.189 INFO:teuthology.orchestra.run.smithi190.stdout: "key": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:39:36.190 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 1014254849 2020-11-09T12:39:36.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:36.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:36.190 INFO:teuthology.orchestra.run.smithi190.stdout: "key": "bd4c3e31-4a1f-445d-9b97-3f53c3f0f6b3", 2020-11-09T12:39:36.190 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 2419824616 2020-11-09T12:39:36.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:36.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:36.191 INFO:teuthology.orchestra.run.smithi190.stdout: "key": "c104565e-7c15-4c5e-9b2c-908de1973930", 2020-11-09T12:39:36.191 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 623289585 2020-11-09T12:39:36.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:36.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:36.191 INFO:teuthology.orchestra.run.smithi190.stdout: "key": "e563eedc-cfbd-426b-a4d4-572da58c3b13", 2020-11-09T12:39:36.191 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 1100490678 2020-11-09T12:39:36.192 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:36.192 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:39:36.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:36.192 INFO:teuthology.orchestra.run.smithi190.stdout: "master_zonegroup": "test-zonegroup", 2020-11-09T12:39:36.192 INFO:teuthology.orchestra.run.smithi190.stdout: "master_zone": "6dfaac43-4458-47cb-abae-96e6cb987c71", 2020-11-09T12:39:36.192 INFO:teuthology.orchestra.run.smithi190.stdout: "period_config": { 2020-11-09T12:39:36.192 INFO:teuthology.orchestra.run.smithi190.stdout: "bucket_quota": { 2020-11-09T12:39:36.193 INFO:teuthology.orchestra.run.smithi190.stdout: "enabled": false, 2020-11-09T12:39:36.193 INFO:teuthology.orchestra.run.smithi190.stdout: "check_on_raw": false, 2020-11-09T12:39:36.193 INFO:teuthology.orchestra.run.smithi190.stdout: "max_size": -1, 2020-11-09T12:39:36.193 INFO:teuthology.orchestra.run.smithi190.stdout: "max_size_kb": 0, 2020-11-09T12:39:36.193 INFO:teuthology.orchestra.run.smithi190.stdout: "max_objects": -1 2020-11-09T12:39:36.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:36.193 INFO:teuthology.orchestra.run.smithi190.stdout: "user_quota": { 2020-11-09T12:39:36.194 INFO:teuthology.orchestra.run.smithi190.stdout: "enabled": false, 2020-11-09T12:39:36.194 INFO:teuthology.orchestra.run.smithi190.stdout: "check_on_raw": false, 2020-11-09T12:39:36.194 INFO:teuthology.orchestra.run.smithi190.stdout: "max_size": -1, 2020-11-09T12:39:36.194 INFO:teuthology.orchestra.run.smithi190.stdout: "max_size_kb": 0, 2020-11-09T12:39:36.194 INFO:teuthology.orchestra.run.smithi190.stdout: "max_objects": -1 2020-11-09T12:39:36.194 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:36.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:36.194 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_id": "32656115-d19e-4547-bae3-a6fce59494ce", 2020-11-09T12:39:36.195 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_name": "test-realm", 2020-11-09T12:39:36.195 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:36.195 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:39:36.196 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-11-09T12:39:36.196 INFO:teuthology.orchestra.run:waiting for 300 2020-11-09T12:39:36.240 INFO:tasks.rgw.c2.client.1.smithi190.stdout:2020-11-09T12:39:36.238+0000 7f4f35b4f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 14541) UID: 0 2020-11-09T12:39:36.240 INFO:tasks.rgw.c2.client.1.smithi190.stdout:2020-11-09T12:39:36.238+0000 7f4f4d5e7100 -1 shutting down 2020-11-09T12:39:39.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-11-09T12:39:42.298 INFO:tasks.rgw.c2.client.1:Stopped 2020-11-09T12:39:42.299 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-11-09T12:39:42.299 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-11-09T12:39:42.302 INFO:tasks.rgw.c2.client.1:Started 2020-11-09T12:39:42.303 INFO:teuthology.orchestra.run.smithi190:> curl http://smithi190.front.sepia.ceph.com:8001 2020-11-09T12:39:42.319 INFO:teuthology.orchestra.run.smithi190.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T12:39:42.319 INFO:teuthology.orchestra.run.smithi190.stderr: Dload Upload Total Spent Left Speed 2020-11-09T12:39:42.322 INFO:teuthology.orchestra.run.smithi190.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi190.front.sepia.ceph.com port 8001: Connection refused 2020-11-09T12:39:42.324 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-09T12:39:43.325 INFO:teuthology.orchestra.run.smithi190:> curl http://smithi190.front.sepia.ceph.com:8001 2020-11-09T12:39:43.342 INFO:teuthology.orchestra.run.smithi190.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-09T12:39:43.342 INFO:teuthology.orchestra.run.smithi190.stderr: Dload Upload Total Spent Left Speed 2020-11-09T12:39:43.346 INFO:teuthology.orchestra.run.smithi190.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 33500 0 --:--:-- --:--:-- --:--:-- 33500 2020-11-09T12:39:43.350 INFO:teuthology.orchestra.run.smithi190.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005fa9388f-10e9-test-zone4","HostId":"10e9-test-zone4-test-zonegroup"} 2020-11-09T12:39:43.350 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-11-09T12:39:43.351 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-11-09T12:39:43.426 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-11-09T12:39:43.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:43.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-09T12:39:43.427 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-09T12:39:43.471 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:39:43.472 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:39:43.679 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:39:43.680 INFO:teuthology.orchestra.run.smithi164.stdout: "user_id": "rgw-multisite-test-user", 2020-11-09T12:39:43.680 INFO:teuthology.orchestra.run.smithi164.stdout: "display_name": "Multisite Test User", 2020-11-09T12:39:43.680 INFO:teuthology.orchestra.run.smithi164.stdout: "email": "", 2020-11-09T12:39:43.680 INFO:teuthology.orchestra.run.smithi164.stdout: "suspended": 0, 2020-11-09T12:39:43.681 INFO:teuthology.orchestra.run.smithi164.stdout: "max_buckets": 1000, 2020-11-09T12:39:43.681 INFO:teuthology.orchestra.run.smithi164.stdout: "subusers": [], 2020-11-09T12:39:43.681 INFO:teuthology.orchestra.run.smithi164.stdout: "keys": [ 2020-11-09T12:39:43.681 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:43.681 INFO:teuthology.orchestra.run.smithi164.stdout: "user": "rgw-multisite-test-user", 2020-11-09T12:39:43.682 INFO:teuthology.orchestra.run.smithi164.stdout: "access_key": "KHPVRI8S76X0HOYR3FDJ", 2020-11-09T12:39:43.682 INFO:teuthology.orchestra.run.smithi164.stdout: "secret_key": "cl4B0YUYQGv27d4Yb9mEHFF9s5LcVhd5vsPBHdeD" 2020-11-09T12:39:43.682 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:39:43.682 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2020-11-09T12:39:43.682 INFO:teuthology.orchestra.run.smithi164.stdout: "swift_keys": [], 2020-11-09T12:39:43.682 INFO:teuthology.orchestra.run.smithi164.stdout: "caps": [], 2020-11-09T12:39:43.683 INFO:teuthology.orchestra.run.smithi164.stdout: "op_mask": "read, write, delete", 2020-11-09T12:39:43.683 INFO:teuthology.orchestra.run.smithi164.stdout: "default_placement": "", 2020-11-09T12:39:43.683 INFO:teuthology.orchestra.run.smithi164.stdout: "default_storage_class": "", 2020-11-09T12:39:43.683 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_tags": [], 2020-11-09T12:39:43.683 INFO:teuthology.orchestra.run.smithi164.stdout: "bucket_quota": { 2020-11-09T12:39:43.684 INFO:teuthology.orchestra.run.smithi164.stdout: "enabled": false, 2020-11-09T12:39:43.684 INFO:teuthology.orchestra.run.smithi164.stdout: "check_on_raw": false, 2020-11-09T12:39:43.684 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size": -1, 2020-11-09T12:39:43.684 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size_kb": 0, 2020-11-09T12:39:43.684 INFO:teuthology.orchestra.run.smithi164.stdout: "max_objects": -1 2020-11-09T12:39:43.684 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:43.685 INFO:teuthology.orchestra.run.smithi164.stdout: "user_quota": { 2020-11-09T12:39:43.685 INFO:teuthology.orchestra.run.smithi164.stdout: "enabled": false, 2020-11-09T12:39:43.685 INFO:teuthology.orchestra.run.smithi164.stdout: "check_on_raw": false, 2020-11-09T12:39:43.685 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size": -1, 2020-11-09T12:39:43.685 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size_kb": 0, 2020-11-09T12:39:43.685 INFO:teuthology.orchestra.run.smithi164.stdout: "max_objects": -1 2020-11-09T12:39:43.686 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:43.686 INFO:teuthology.orchestra.run.smithi164.stdout: "temp_url_keys": [], 2020-11-09T12:39:43.686 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "rgw", 2020-11-09T12:39:43.686 INFO:teuthology.orchestra.run.smithi164.stdout: "mfa_ids": [] 2020-11-09T12:39:43.686 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:39:43.686 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-11-09T12:39:43.709 INFO:rgw_multi.tests:meta checkpoint 2020-11-09T12:39:43.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:39:43.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:39:43.710 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:39:43.752 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:39:43.752 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:39:43.953 INFO:teuthology.orchestra.run.smithi164.stderr:No --period given, using current period=d8c95da7-75a8-4dcf-94ba-f12d5c80a46f 2020-11-09T12:39:43.988 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:39:43.990 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:43.990 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:39:43.990 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:39:43.671316Z" 2020-11-09T12:39:43.991 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:43.991 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:43.991 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:39:43.993 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:39:43.994 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:43.994 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:43.995 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:39:43.995 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:39:43.995 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:39:43.995 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:39:43.996 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:39:43.996 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:39:43.996 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:39:43.997 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:39:43.999 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T12:39:43.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:39:44.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:39:44.000 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:39:44.042 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:39:44.042 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:39:44.290 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:39:44.290 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:39:44.291 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:39:44.291 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:39:44.291 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:39:44.291 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:39:44.291 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:44.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:44.292 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:39:44.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:44.292 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:39:44.292 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:39:44.293 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:39:44.293 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:39:44.293 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:39:44.293 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:39:44.293 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:39:44.294 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:39:44.294 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:44.294 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:44.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:44.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:44.294 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:39:44.294 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:39:44.295 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:39:44.295 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:39:44.295 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:39:44.295 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:39:44.295 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:39:44.295 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:39:44.295 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:44.295 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:44.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:44.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:44.296 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:39:44.296 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:39:44.296 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:39:44.296 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:39:44.297 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:39:44.297 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:39:44.297 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:39:44.297 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:39:44.297 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:44.297 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:44.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:44.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:44.298 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:39:44.298 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:39:44.298 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:39:44.298 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:39:44.298 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:39:44.298 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:39:44.299 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:39:44.299 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:39:44.299 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:44.299 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:44.299 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:44.300 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:39:44.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:44.300 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:39:44.300 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:39:44.300 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:39:44.300 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:44.300 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:39:44.301 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:39:44.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2020-11-09T12:39:49.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:39:49.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:39:49.304 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:39:49.345 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:39:49.346 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:39:49.597 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:39:49.598 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:39:49.598 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:39:49.598 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:39:49.598 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:39:49.598 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:39:49.599 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:49.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:49.599 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:39:49.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:49.599 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:39:49.599 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:39:49.599 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:39:49.600 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:39:49.600 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:39:49.600 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:39:49.600 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:39:49.600 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:39:49.600 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:49.600 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:49.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:49.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:49.601 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:39:49.601 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:39:49.601 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:39:49.601 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:39:49.602 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:39:49.602 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:39:49.602 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:39:49.602 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:39:49.602 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:49.602 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:49.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:49.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:49.603 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:39:49.603 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:39:49.603 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:39:49.603 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:39:49.603 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:39:49.603 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:39:49.604 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:39:49.604 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:39:49.604 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:49.604 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:49.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:49.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:49.604 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:39:49.605 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:39:49.605 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:39:49.605 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:39:49.605 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:39:49.605 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:39:49.605 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:39:49.605 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:39:49.606 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:49.606 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:49.606 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:49.606 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:39:49.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:49.606 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:39:49.607 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:39:49.607 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:39:49.607 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:49.607 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:39:49.608 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:39:54.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:39:54.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:39:54.611 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:39:54.652 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:39:54.652 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:39:54.902 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:39:54.902 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:39:54.903 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:39:54.903 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:39:54.903 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:39:54.903 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:39:54.903 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:54.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:54.903 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:39:54.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:54.904 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:39:54.904 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:39:54.904 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:39:54.904 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:39:54.904 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:39:54.904 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:39:54.905 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:39:54.905 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:39:54.905 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:54.905 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:54.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:54.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:54.905 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:39:54.906 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:39:54.906 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:39:54.906 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:39:54.906 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:39:54.906 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:39:54.906 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:39:54.906 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:39:54.907 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:54.907 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:54.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:54.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:54.907 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:39:54.907 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:39:54.907 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:39:54.907 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:39:54.908 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:39:54.908 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:39:54.908 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:39:54.908 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:39:54.908 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:54.908 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:54.908 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:54.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:39:54.909 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:39:54.909 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:39:54.909 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:39:54.909 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:39:54.909 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:39:54.909 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:39:54.909 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:39:54.910 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:39:54.910 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:39:54.910 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:54.910 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:54.910 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:39:54.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:39:54.910 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:39:54.911 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:39:54.911 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:39:54.911 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:39:54.911 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:39:54.912 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:39:59.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:39:59.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:39:59.914 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:39:59.956 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:39:59.957 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:40:00.202 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:40:00.202 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:40:00.202 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:40:00.203 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:40:00.203 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:40:00.203 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:40:00.203 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:00.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:00.203 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:40:00.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:00.204 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:40:00.204 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:00.204 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:00.204 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:00.204 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:00.204 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:40:00.205 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:00.205 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:00.205 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:00.205 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:00.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:00.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:00.206 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:40:00.206 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:00.206 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:00.207 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:00.207 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:00.207 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:00.207 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:00.207 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:00.207 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:00.207 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:00.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:00.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:00.208 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:40:00.208 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:00.208 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:00.208 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:00.208 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:00.208 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:00.209 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:00.209 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:00.209 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:00.209 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:00.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:00.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:00.209 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:40:00.210 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:00.210 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:00.210 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:00.210 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:00.210 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:00.210 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:00.210 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:00.211 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:00.211 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:00.211 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:00.211 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:40:00.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:00.211 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:40:00.211 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:40:00.211 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:40:00.212 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:00.212 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:40:00.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:40:05.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:05.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:05.215 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:40:05.255 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:40:05.256 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:40:05.499 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:40:05.500 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:40:05.500 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:40:05.500 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:40:05.500 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:40:05.500 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:40:05.501 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:05.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:05.501 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:40:05.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:05.501 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:40:05.501 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:05.502 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:05.502 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:05.502 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:05.502 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:40:05.502 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:05.502 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:05.502 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:05.503 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:05.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:05.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:05.503 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:40:05.503 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:05.503 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:05.503 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:05.504 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:05.504 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:05.504 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:05.504 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:05.504 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:05.504 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:05.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:05.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:05.505 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:40:05.505 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:05.505 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:05.505 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:05.505 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:05.505 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:05.506 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:05.507 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:05.507 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:05.507 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:05.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:05.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:05.507 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:40:05.507 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:05.508 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:05.508 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:05.508 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:05.508 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:05.508 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:05.508 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:05.508 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:05.509 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:05.509 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:05.509 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:40:05.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:05.509 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:40:05.509 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:40:05.509 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:40:05.510 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:05.510 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:40:05.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:40:10.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:10.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:10.513 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:40:10.554 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:40:10.555 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:40:10.808 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:40:10.808 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:40:10.808 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:40:10.808 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:40:10.808 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:40:10.809 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:40:10.809 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:10.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:10.809 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:40:10.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:10.809 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:40:10.809 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:10.810 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:10.810 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:10.810 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:10.810 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:40:10.810 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:10.810 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:10.810 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:10.811 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:10.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:10.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:10.811 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:40:10.811 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:10.811 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:10.811 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:10.812 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:10.812 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:10.812 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:10.812 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:10.812 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:10.812 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:10.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:10.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:10.813 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:40:10.813 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:10.813 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:10.813 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:10.813 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:10.813 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:10.813 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:10.814 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:10.814 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:10.814 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:10.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:10.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:10.814 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:40:10.814 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:10.815 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:10.815 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:10.815 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:10.815 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:10.815 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:10.815 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:10.815 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:10.815 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:10.816 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:10.816 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:40:10.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:10.816 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:40:10.816 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:40:10.816 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:40:10.816 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:10.817 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:40:10.818 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:40:15.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:15.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:15.820 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:40:15.856 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:40:15.856 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:40:16.107 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:40:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:40:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:40:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:40:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:40:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:40:16.108 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:40:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:40:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:16.109 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:40:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:16.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:40:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:16.111 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:40:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:40:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:16.114 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:16.115 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:40:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:40:16.116 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:40:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:40:16.117 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:16.117 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:40:16.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:40:21.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:21.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:21.120 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:40:21.161 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:40:21.161 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:40:21.399 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:40:21.400 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:40:21.400 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:40:21.400 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:40:21.400 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:40:21.400 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:40:21.400 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:21.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:21.401 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:40:21.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:21.401 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:40:21.401 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:21.401 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:21.401 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:21.402 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:21.402 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:40:21.402 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:21.402 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:21.402 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:21.402 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:21.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:21.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:21.403 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:40:21.403 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:21.403 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:21.403 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:21.403 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:21.403 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:21.404 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:21.404 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:21.404 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:21.404 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:21.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:21.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:21.404 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:40:21.404 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:21.405 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:21.405 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:21.405 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:21.405 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:21.405 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:21.405 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:21.405 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:21.406 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:21.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:21.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:21.406 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:40:21.406 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:21.406 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:21.406 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:21.407 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:21.407 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:21.407 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:21.407 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:21.407 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:21.407 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:21.407 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:21.408 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:40:21.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:21.408 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:40:21.408 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:40:21.408 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:40:21.408 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:21.408 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:40:21.409 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:40:26.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:26.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:26.411 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:40:26.451 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:40:26.451 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:40:26.719 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:40:26.719 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:40:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:40:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:40:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:40:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:40:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:26.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:40:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:40:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:26.721 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:40:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:26.722 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:40:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:26.723 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:40:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:26.724 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:26.725 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:40:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:26.726 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:26.727 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:40:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:40:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:40:26.728 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:40:26.729 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:26.729 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:40:26.730 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:40:31.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:31.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:31.732 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:40:31.777 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:40:31.777 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:40:31.986 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:40:31.986 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:40:31.986 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:40:31.986 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:40:31.987 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:40:31.987 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:40:31.987 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:31.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:31.987 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:40:31.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:31.987 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:40:31.988 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:31.988 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:31.988 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:31.989 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:31.990 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:40:31.990 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:31.990 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:40:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:31.994 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:31.994 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:31.994 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:31.995 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:31.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:31.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:31.995 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:40:31.996 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:31.996 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:31.996 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:31.996 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:31.997 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:31.997 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:31.997 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:31.998 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:31.998 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:31.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:31.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:31.999 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:40:31.999 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:31.999 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:32.000 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:32.000 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:32.000 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:32.001 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:32.001 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:32.001 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:32.001 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:32.002 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:32.002 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:40:32.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:32.002 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:40:32.003 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:40:32.003 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:40:32.003 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:32.004 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:40:32.005 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:40:37.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:37.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:37.008 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:40:37.051 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:40:37.051 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:40:37.300 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:40:37.300 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:40:37.300 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:40:37.300 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:40:37.301 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:40:37.301 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:40:37.301 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:37.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:37.301 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:40:37.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:37.302 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:40:37.302 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:37.302 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:37.302 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:37.302 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:37.303 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:40:37.303 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:37.303 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:37.303 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:37.303 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:37.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:37.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:37.304 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:40:37.304 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:37.304 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:37.304 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:37.304 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:37.304 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:37.305 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:37.305 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:37.305 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:37.305 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:37.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:37.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:37.306 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:40:37.306 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:37.306 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:37.306 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:37.306 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:37.306 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:37.306 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:37.307 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:37.307 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:37.307 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:37.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:37.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:37.307 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:40:37.308 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:37.308 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:37.308 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:37.308 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:37.308 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:37.308 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:37.308 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:37.308 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:37.309 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:37.309 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:37.309 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:40:37.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:37.309 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:40:37.309 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:40:37.310 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:40:37.310 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:37.310 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:40:37.311 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:40:42.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:42.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:42.314 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:40:42.354 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:40:42.355 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:40:42.598 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:40:42.599 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:40:42.599 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:40:42.599 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:40:42.599 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:40:42.600 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:40:42.600 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:42.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:42.600 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:40:42.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:42.600 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:40:42.601 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:42.601 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:42.601 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:42.601 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:42.601 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:40:42.601 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:42.602 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:42.602 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:42.602 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:42.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:42.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:42.602 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:40:42.603 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:42.603 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:42.603 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:42.603 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:42.603 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:42.603 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:42.604 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:42.604 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:42.604 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:42.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:42.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:42.604 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:40:42.605 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:42.605 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:42.605 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:42.605 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:42.605 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:42.605 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:42.606 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:42.606 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:42.606 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:42.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:42.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:42.606 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:40:42.606 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:42.607 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:42.607 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:42.607 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:42.607 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:42.607 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:42.607 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:42.608 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:42.608 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:42.608 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:42.608 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:40:42.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:42.608 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:40:42.609 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:40:42.609 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:40:42.609 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:42.609 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:40:42.610 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:40:47.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:47.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:47.613 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:40:47.654 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:40:47.654 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:40:47.890 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:40:47.890 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:40:47.890 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:40:47.890 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:40:47.891 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:40:47.891 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:40:47.891 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:47.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:47.891 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:40:47.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:47.891 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:40:47.891 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:47.892 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:47.892 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:47.892 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:47.892 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:40:47.892 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:47.892 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:47.893 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:47.893 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:47.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:47.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:47.893 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:40:47.893 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:47.894 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:47.894 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:47.894 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:47.894 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:47.894 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:47.894 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:47.895 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:47.895 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:47.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:47.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:47.895 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:40:47.895 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:47.896 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:47.896 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:47.896 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:47.896 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:47.896 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:47.896 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:47.896 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:47.897 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:47.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:47.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:47.897 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:40:47.897 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:47.897 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:47.897 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:47.898 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:47.898 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:47.898 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:47.898 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:47.898 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:47.898 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:47.898 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:47.898 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:40:47.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:47.899 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:40:47.899 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:40:47.899 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:40:47.899 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:47.899 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:40:47.900 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:40:52.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:52.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:52.902 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:40:52.942 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:40:52.942 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:40:53.203 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:40:53.204 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:40:53.204 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:40:53.204 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:40:53.204 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:40:53.204 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:40:53.205 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:53.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:53.205 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:40:53.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:53.205 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:40:53.205 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:53.205 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:53.206 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:53.206 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:53.206 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:40:53.206 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:53.206 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:53.206 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:53.207 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:53.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:53.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:53.207 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:40:53.207 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:53.207 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:53.207 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:53.207 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:53.208 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:53.208 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:53.208 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:53.208 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:53.208 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:53.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:53.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:53.209 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:40:53.209 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:53.209 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:53.209 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:53.209 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:53.209 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:53.209 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:53.209 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:53.210 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:53.210 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:53.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:53.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:53.210 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:40:53.210 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:53.210 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:53.211 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:53.211 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:53.211 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:53.211 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:53.211 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:53.211 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:53.211 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:53.212 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:53.212 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:40:53.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:53.212 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:40:53.212 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:40:53.212 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:40:53.212 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:53.212 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:40:53.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:40:58.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:58.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:40:58.215 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:40:58.254 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:40:58.255 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:40:58.513 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:40:58.514 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:40:58.514 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:40:58.514 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:40:58.514 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:40:58.514 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:40:58.514 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:58.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:58.515 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:40:58.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:58.515 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:40:58.515 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:58.515 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:58.515 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:58.515 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:58.516 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:40:58.516 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:58.516 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:58.516 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:58.516 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:58.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:58.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:58.517 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:40:58.517 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:58.517 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:58.517 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:58.517 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:58.517 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:58.517 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:58.518 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:58.518 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:58.518 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:58.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:58.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:58.518 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:40:58.518 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:58.519 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:58.519 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:58.519 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:58.519 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:58.519 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:58.519 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:58.519 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:58.519 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:58.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:58.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:40:58.520 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:40:58.520 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:40:58.520 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:40:58.520 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:40:58.520 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:40:58.521 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:40:58.521 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:40:58.521 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:40:58.521 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:40:58.521 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:58.521 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:58.521 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:40:58.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:40:58.522 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:40:58.522 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:40:58.522 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:40:58.522 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:40:58.522 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:40:58.523 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:41:03.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:03.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:03.525 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:03.567 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:03.568 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:03.792 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:03.793 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:03.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:03.794 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:03.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:03.794 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:03.794 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:03.794 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:03.795 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:03.795 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:03.795 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:03.795 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:03.795 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:03.795 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:03.795 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:03.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:03.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:03.796 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:03.796 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:03.796 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:03.796 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:03.796 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:03.797 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:03.797 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:03.797 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:03.797 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:03.797 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:03.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:03.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:03.798 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:03.798 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:03.798 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:03.798 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:03.798 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:03.798 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:03.798 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:03.799 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:03.799 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:03.799 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:03.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:03.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:03.799 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:03.799 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:03.800 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:03.800 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:03.800 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:03.800 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:03.800 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:03.800 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:03.800 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:03.801 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:03.801 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:03.801 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:03.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:03.801 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:03.801 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:03.801 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:03.802 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:03.802 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:03.803 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:41:08.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:08.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:08.805 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:08.847 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:08.847 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:09.090 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:09.090 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:09.091 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:09.091 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:09.091 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:09.091 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:09.091 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:09.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:09.092 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:09.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:09.092 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:09.092 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:09.092 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:09.092 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:09.092 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:09.093 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:09.093 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:09.093 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:09.093 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:09.093 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:09.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:09.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:09.093 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:09.094 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:09.094 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:09.094 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:09.094 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:09.094 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:09.094 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:09.094 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:09.095 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:09.095 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:09.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:09.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:09.095 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:09.095 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:09.095 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:09.096 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:09.096 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:09.096 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:09.096 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:09.096 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:09.096 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:09.096 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:09.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:09.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:09.097 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:09.097 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:09.097 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:09.097 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:09.097 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:09.097 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:09.098 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:09.098 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:09.098 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:09.098 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:09.098 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:09.098 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:09.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:09.099 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:09.099 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:09.099 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:09.099 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:09.099 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:09.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:41:14.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:14.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:14.102 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:14.143 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:14.144 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:14.411 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:14.412 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:14.412 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:14.412 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:14.412 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:14.413 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:14.413 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:14.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:14.413 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:14.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:14.413 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:14.413 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:14.414 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:14.414 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:14.414 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:14.414 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:14.414 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:14.414 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:14.414 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:14.415 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:14.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:14.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:14.415 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:14.415 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:14.415 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:14.415 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:14.416 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:14.416 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:14.416 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:14.416 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:14.416 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:14.416 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:14.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:14.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:14.417 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:14.417 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:14.417 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:14.417 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:14.417 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:14.417 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:14.418 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:14.418 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:14.418 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:14.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:14.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:14.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:14.418 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:14.419 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:14.419 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:14.419 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:14.419 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:14.419 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:14.419 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:14.419 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:14.419 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:14.420 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:14.420 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:14.420 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:14.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:14.420 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:14.420 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:14.420 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:14.421 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:14.421 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:14.422 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:41:19.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:19.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:19.425 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:19.466 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:19.467 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:19.704 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:19.704 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:19.708 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:19.708 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:19.708 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:19.708 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:19.708 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:19.708 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:19.708 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:19.709 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:19.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:19.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:19.709 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:19.709 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:19.709 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:19.709 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:19.710 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:19.710 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:19.710 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:19.710 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:19.710 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:19.710 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:19.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:19.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:19.711 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:19.711 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:19.711 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:19.711 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:19.711 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:19.711 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:19.711 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:19.712 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:19.712 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:19.712 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:19.712 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:19.712 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:19.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:19.712 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:19.713 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:19.713 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:19.713 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:19.713 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:19.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:41:24.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:24.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:24.717 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:24.758 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:24.758 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:25.020 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:25.021 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:25.021 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:25.021 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:25.021 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:25.021 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:25.021 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:25.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:25.022 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:25.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:25.022 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:25.022 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:25.022 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:25.022 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:25.023 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:25.023 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:25.023 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:25.023 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:25.023 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:25.023 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:25.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:25.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:25.024 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:25.024 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:25.024 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:25.024 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:25.024 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:25.024 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:25.024 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:25.025 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:25.025 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:25.025 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:25.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:25.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:25.025 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:25.025 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:25.025 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:25.026 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:25.026 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:25.026 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:25.026 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:25.026 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:25.026 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:25.026 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:25.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:25.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:25.027 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:25.027 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:25.027 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:25.027 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:25.027 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:25.028 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:25.028 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:25.028 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:25.028 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:25.028 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:25.028 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:25.028 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:25.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:25.029 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:25.029 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:25.029 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:25.029 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:25.029 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:25.030 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:41:30.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:30.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:30.033 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:30.076 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:30.077 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:30.330 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:30.331 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:30.331 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:30.331 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:30.331 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:30.331 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:30.331 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:30.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:30.332 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:30.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:30.332 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:30.332 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:30.332 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:30.332 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:30.332 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:30.333 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:30.333 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:30.333 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:30.333 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:30.333 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:30.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:30.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:30.334 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:30.334 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:30.334 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:30.334 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:30.334 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:30.334 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:30.334 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:30.335 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:30.335 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:30.335 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:30.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:30.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:30.335 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:30.335 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:30.336 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:30.336 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:30.336 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:30.336 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:30.336 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:30.336 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:30.336 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:30.336 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:30.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:30.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:30.337 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:30.337 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:30.337 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:30.337 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:30.337 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:30.338 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:30.338 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:30.338 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:30.338 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:30.338 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:30.338 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:30.338 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:30.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:30.339 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:30.339 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:30.339 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:30.339 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:30.339 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:30.340 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:41:35.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:35.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:35.343 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:35.384 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:35.384 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:35.628 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:35.628 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:35.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:35.632 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:35.633 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:35.634 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:35.635 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:35.636 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:35.637 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:35.638 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:35.638 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:35.639 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1604925583.671316_42.1 target= 2020-11-09T12:41:40.640 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:40.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:40.641 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:40.681 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:40.681 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:40.939 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:40.940 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:40.940 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:40.940 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:40.940 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:40.940 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:40.940 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:40.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:40.941 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:40.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:40.941 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:40.941 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:40.941 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:40.941 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:40.942 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:40.942 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:40.942 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:40.942 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:41:40.942 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:40.942 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:40.942 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:40.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:40.943 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:40.943 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:40.943 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:40.943 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:40.943 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:40.943 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:40.944 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:40.944 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:40.944 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:40.944 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:40.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:40.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:40.944 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:40.945 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:40.945 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:40.945 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:40.945 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:40.945 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:40.945 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:40.945 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:40.945 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:40.946 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:40.946 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:40.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:40.946 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:40.946 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:40.946 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:40.946 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:40.947 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:40.947 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:40.947 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:40.947 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:40.947 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:40.947 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:40.947 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:40.947 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:40.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:40.948 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:40.948 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:40.948 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:40.948 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:40.948 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:40.949 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T12:41:40.950 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T12:41:40.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:40.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:40.951 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:40.994 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:40.994 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:41.250 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:41:41.250 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:41:41.251 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:41:41.251 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:41:41.251 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:41:41.251 INFO:teuthology.orchestra.run.smithi164.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:41.251 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:41.251 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:41.251 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:41:41.252 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:41.252 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:41:41.252 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:41.252 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:41.252 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:41.252 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:41.252 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 1, 2020-11-09T12:41:41.253 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:41.253 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:41:41.253 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:41.253 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:41.253 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:41.253 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:41.253 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:41:41.254 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:41.254 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:41.254 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:41.254 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:41.254 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:41:41.254 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:41.254 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:41:41.255 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:41.255 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:41.255 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:41.255 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:41.255 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:41:41.255 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:41.255 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:41.255 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:41.256 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:41.256 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:41:41.256 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:41.256 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:41:41.256 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:41.256 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:41.256 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:41.257 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:41.257 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:41:41.257 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:41.257 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:41.257 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:41.258 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:41.258 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:41:41.258 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:41.258 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:41:41.258 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:41.258 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:41.258 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:41.259 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:41:41.259 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:41.259 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:41:41.259 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 1, 2020-11-09T12:41:41.259 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 1 2020-11-09T12:41:41.259 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:41.259 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:41:41.260 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T12:41:41.261 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T12:41:41.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:41.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:41.262 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:41.303 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:41.303 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:41.564 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:41.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:41.566 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:41:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:41.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:41.568 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:41.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:41.570 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:41.571 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:41.572 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:41.573 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:41.574 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:41.574 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:41.575 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T12:41:41.575 INFO:rgw_multi.tests:meta checkpoint 2020-11-09T12:41:41.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:41.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:41.576 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:41.614 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:41.615 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:41.826 INFO:teuthology.orchestra.run.smithi164.stderr:No --period given, using current period=d8c95da7-75a8-4dcf-94ba-f12d5c80a46f 2020-11-09T12:41:41.857 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:41:41.857 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:41.857 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:39:43.671316Z" 2020-11-09T12:41:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:41:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:41.859 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:41.859 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:41:41.859 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:41.859 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:41.859 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:41.859 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:41:41.859 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:41.860 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:41:41.861 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T12:41:41.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:41.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:41.862 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:41.901 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:41.902 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:42.140 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:42.140 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:42.140 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:42.140 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:42.141 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:42.141 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:42.141 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:42.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:42.141 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:42.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:42.141 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:42.142 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:42.142 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:42.142 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:42.142 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:42.142 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:42.142 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:42.143 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:41:42.143 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:42.143 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:42.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:42.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:42.143 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:42.143 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:42.143 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:42.144 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:42.144 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:42.144 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:42.144 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:42.144 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:42.144 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:42.144 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:42.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:42.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:42.145 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:42.145 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:42.145 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:42.145 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:42.145 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:42.146 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:42.146 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:42.146 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:42.146 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:42.146 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:42.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:42.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:42.147 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:42.148 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:42.149 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:42.149 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:42.150 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T12:41:42.150 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T12:41:42.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:42.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:42.151 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:42.190 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:42.191 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:42.439 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:41:42.439 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:41:42.439 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:41:42.439 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:41:42.440 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:41:42.440 INFO:teuthology.orchestra.run.smithi164.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:42.440 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:42.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:42.440 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:41:42.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:42.440 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:41:42.441 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:42.441 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:42.441 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:42.441 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:42.441 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 1, 2020-11-09T12:41:42.441 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:42.441 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:41:42.442 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:42.442 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:42.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:42.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:42.442 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:41:42.442 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:42.442 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:42.443 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:42.443 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:42.443 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:41:42.443 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:42.443 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:41:42.443 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:42.443 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:42.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:42.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:42.444 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:41:42.444 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:42.444 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:42.444 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:42.444 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:42.444 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:41:42.445 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:42.445 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:41:42.445 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:42.445 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:42.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:42.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:42.445 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:41:42.445 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:42.446 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:42.446 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:42.446 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:42.446 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:41:42.446 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:42.446 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:41:42.446 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:42.447 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:42.447 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:42.447 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:41:42.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:42.447 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:41:42.447 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 1, 2020-11-09T12:41:42.447 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 1 2020-11-09T12:41:42.448 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:42.448 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:41:42.449 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T12:41:42.449 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T12:41:42.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:42.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:42.451 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:42.491 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:42.492 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:42.755 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:42.755 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:42.756 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:42.756 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:42.756 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:42.756 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:42.756 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:42.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:42.756 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:42.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:42.757 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:42.757 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:42.757 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:42.757 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:42.757 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:42.757 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:42.758 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:42.758 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:41:42.758 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:42.758 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:42.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:42.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:42.758 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:42.759 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:42.759 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:42.759 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:42.759 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:42.759 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:42.759 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:42.759 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:42.759 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:42.760 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:42.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:42.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:42.760 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:42.760 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:42.760 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:42.760 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:42.761 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:42.761 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:42.761 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:42.761 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:42.761 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:42.761 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:42.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:42.762 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:42.762 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:42.762 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:42.762 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:42.762 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:42.762 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:42.762 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:42.763 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:42.763 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:42.763 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:42.763 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:42.763 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:42.763 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:42.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:42.763 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:42.764 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:42.764 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:42.764 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:42.764 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:42.765 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T12:41:42.765 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-11-09T12:41:42.776 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kuljus-1 2020-11-09T12:41:46.094 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kuljus-2 2020-11-09T12:41:49.881 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kuljus-3 2020-11-09T12:41:49.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:49.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:49.947 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:49.982 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:49.983 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:50.204 INFO:teuthology.orchestra.run.smithi164.stderr:No --period given, using current period=d8c95da7-75a8-4dcf-94ba-f12d5c80a46f 2020-11-09T12:41:50.239 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:41:50.240 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:50.240 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:50.240 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:39:43.671316Z" 2020-11-09T12:41:50.240 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:50.240 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:50.240 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925709.923492_84.1", 2020-11-09T12:41:50.241 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:41:49.923492Z" 2020-11-09T12:41:50.241 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:50.241 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:50.241 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:50.241 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:41:50.241 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:50.241 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:50.242 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:50.242 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:41:50.242 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:50.242 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:41:50.243 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T12:41:50.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:50.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:50.244 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:50.285 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:50.285 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:50.545 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:50.546 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:50.546 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:50.546 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:50.546 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:50.546 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:50.547 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:50.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:50.547 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:50.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:50.547 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:50.547 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:50.548 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:50.548 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:50.548 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:50.548 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:50.548 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:50.548 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:41:50.548 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:50.549 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:50.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:50.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:50.549 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:50.549 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:50.549 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:50.549 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925709.923492_84.1", 2020-11-09T12:41:50.550 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:50.550 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:50.550 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:50.550 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:41:49.923492Z", 2020-11-09T12:41:50.550 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:50.550 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:50.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:50.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:50.551 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:50.551 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:50.551 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:50.551 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:50.551 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:50.551 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:50.552 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:50.552 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:50.552 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:50.552 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:50.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:50.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:50.552 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:50.553 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:50.553 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:50.553 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:50.553 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:50.553 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:50.553 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:50.553 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:50.554 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:50.554 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:50.554 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:50.554 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:50.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:50.554 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:50.555 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:50.555 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:50.555 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:50.555 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:50.556 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T12:41:50.556 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T12:41:50.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:50.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:50.558 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:50.599 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:50.600 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:50.854 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:41:50.854 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:41:50.854 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:41:50.855 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:41:50.855 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:41:50.855 INFO:teuthology.orchestra.run.smithi164.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:50.855 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:50.855 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:50.855 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:41:50.856 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:50.856 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:41:50.856 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:50.856 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:50.856 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:50.856 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:50.856 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 1, 2020-11-09T12:41:50.857 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:50.857 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:41:50.857 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:50.857 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:50.857 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:50.857 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:50.857 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:41:50.858 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:50.858 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:50.858 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925709.923492_84.1", 2020-11-09T12:41:50.858 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:50.858 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:41:50.858 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:50.858 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:41:49.923492Z", 2020-11-09T12:41:50.859 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:50.859 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:50.859 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:50.859 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:50.859 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:41:50.859 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:50.859 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:50.860 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:50.860 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:50.860 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:41:50.860 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:50.860 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:41:50.860 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:50.860 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:50.861 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:50.861 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:50.861 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:41:50.861 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:50.861 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:50.861 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:50.862 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:50.862 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:41:50.862 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:50.862 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:41:50.862 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:50.862 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:50.863 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:50.863 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:41:50.863 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:50.863 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:41:50.863 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 1, 2020-11-09T12:41:50.863 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 1 2020-11-09T12:41:50.863 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:50.864 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:41:50.865 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T12:41:50.865 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T12:41:50.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:50.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:50.866 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:50.906 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:50.906 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:51.166 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:51.167 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:51.167 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:51.167 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:51.167 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:51.167 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:51.167 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:51.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:51.168 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:51.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:51.168 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:51.168 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:51.168 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:51.168 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:51.169 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:51.169 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:51.169 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:51.169 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:41:51.169 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:51.169 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:51.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:51.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:51.170 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:51.170 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:51.170 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:51.170 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925709.923492_84.1", 2020-11-09T12:41:51.170 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:51.170 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:51.171 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:51.171 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:41:49.923492Z", 2020-11-09T12:41:51.171 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:51.171 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:51.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:51.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:51.171 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:51.172 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:51.172 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:51.172 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:51.172 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:51.172 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:51.172 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:51.172 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:51.173 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:51.173 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:51.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:51.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:51.173 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:51.173 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:51.173 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:51.174 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:51.175 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:51.176 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:51.177 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T12:41:52.416 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-11-09T12:41:52.418 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kuljus-4 2020-11-09T12:41:52.452 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kuljus-5 2020-11-09T12:41:52.511 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kuljus-6 2020-11-09T12:41:52.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:52.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:52.589 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:52.629 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:52.630 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:52.849 INFO:teuthology.orchestra.run.smithi164.stderr:No --period given, using current period=d8c95da7-75a8-4dcf-94ba-f12d5c80a46f 2020-11-09T12:41:52.883 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:41:52.883 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:52.884 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:52.884 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:39:43.671316Z" 2020-11-09T12:41:52.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:52.884 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:52.884 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925712.548101_97.1", 2020-11-09T12:41:52.884 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:41:52.548101Z" 2020-11-09T12:41:52.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:52.885 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:52.885 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:52.885 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:41:52.885 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:52.885 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:52.885 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:52.885 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:41:52.886 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:52.886 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:41:52.887 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T12:41:52.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:52.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:52.888 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:52.932 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:52.932 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:53.178 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:53.178 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:53.179 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:53.179 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:53.179 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:53.180 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:53.180 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:53.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:53.181 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:53.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:53.181 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:53.181 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:53.182 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:53.182 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:53.182 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:53.183 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:53.183 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:53.183 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:41:53.184 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:53.184 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:53.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:53.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:53.185 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:53.185 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:53.186 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:53.186 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925712.540270_95.1", 2020-11-09T12:41:53.187 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:53.187 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:53.187 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:53.188 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:41:52.540270Z", 2020-11-09T12:41:53.188 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:53.188 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:53.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:53.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:53.189 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:53.189 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:53.190 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:53.190 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:53.190 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:53.191 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:53.191 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:53.191 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:53.192 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:53.192 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:53.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:53.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:53.193 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:53.193 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:53.193 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:53.193 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:53.194 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:53.194 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:53.194 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:53.194 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:53.195 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:53.195 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:53.195 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:53.195 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:53.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:53.196 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:53.196 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:53.197 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:53.197 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:53.197 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:53.199 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1604925712.548101_97.1 target=1_1604925712.540270_95.1 2020-11-09T12:41:58.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:58.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:58.201 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:58.242 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:58.243 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:58.502 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:58.503 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:58.503 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:58.503 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:58.503 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:58.503 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:58.503 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:58.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:58.504 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:58.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:58.504 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:58.504 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:58.504 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:58.504 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:58.504 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:58.505 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:58.505 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:58.505 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:41:58.505 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:58.505 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:58.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:58.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:58.506 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:58.506 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:58.506 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:58.506 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925712.548101_97.1", 2020-11-09T12:41:58.506 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:58.506 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:58.506 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:58.507 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:41:52.548101Z", 2020-11-09T12:41:58.507 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:58.507 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:58.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:58.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:58.507 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:58.507 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:58.507 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:58.508 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:58.508 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:58.508 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:58.508 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:58.508 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:58.508 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:58.508 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:58.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:58.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:58.509 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:58.509 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:58.509 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:58.509 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:58.509 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:58.509 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:58.510 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:58.510 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:58.510 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:58.510 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:58.510 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:58.510 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:58.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:58.511 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:58.511 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:58.511 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:58.511 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:58.511 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:58.512 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T12:41:58.513 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T12:41:58.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:58.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:58.514 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:58.553 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:58.554 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:58.806 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:41:58.807 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:41:58.807 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:41:58.807 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:41:58.807 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:41:58.807 INFO:teuthology.orchestra.run.smithi164.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:58.807 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:58.808 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:58.808 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:41:58.808 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:58.808 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:41:58.808 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:58.808 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:58.808 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:58.809 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:58.809 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 1, 2020-11-09T12:41:58.809 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:58.809 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:41:58.809 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:58.809 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:58.810 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:58.810 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:58.810 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:41:58.810 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:58.810 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:58.810 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925712.548101_97.1", 2020-11-09T12:41:58.810 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:58.811 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:41:58.811 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:58.811 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:41:52.548101Z", 2020-11-09T12:41:58.811 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:58.811 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:58.811 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:58.811 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:58.811 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:41:58.812 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:58.812 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:58.812 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:58.812 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:58.812 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:41:58.812 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:58.812 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:41:58.813 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:58.813 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:58.813 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:58.813 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:41:58.813 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:41:58.813 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:41:58.813 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:41:58.813 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:41:58.814 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:41:58.814 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:41:58.814 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:41:58.814 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:41:58.814 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:41:58.814 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:58.814 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:58.815 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:41:58.815 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:41:58.815 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:41:58.815 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 1, 2020-11-09T12:41:58.815 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 1 2020-11-09T12:41:58.815 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:41:58.815 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:41:58.816 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T12:41:58.817 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T12:41:58.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:58.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:41:58.818 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:41:58.857 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:41:58.858 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:41:59.124 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:41:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:41:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:41:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:41:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:41:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:41:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:41:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:41:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:41:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:41:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:41:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:41:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925712.540270_95.1", 2020-11-09T12:41:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:41:52.540270Z", 2020-11-09T12:41:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:41:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:41:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:41:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:41:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:41:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:41:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:41:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:41:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:41:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:41:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:41:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:41:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:41:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:41:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:41:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:41:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:41:59.134 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:41:59.134 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1604925712.548101_97.1 target=1_1604925712.540270_95.1 2020-11-09T12:42:04.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:04.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:04.137 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:04.178 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:04.179 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:04.444 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:42:04.445 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:42:04.445 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:42:04.445 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:42:04.445 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:42:04.445 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:04.445 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:04.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:04.446 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:42:04.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:04.446 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:42:04.446 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:04.446 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:04.447 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:42:04.447 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:04.447 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:42:04.447 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:04.447 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:42:04.447 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:04.447 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:04.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:04.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:04.448 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:42:04.448 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:04.448 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:04.448 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925712.548101_97.1", 2020-11-09T12:42:04.448 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:04.448 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:04.449 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:04.449 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:41:52.548101Z", 2020-11-09T12:42:04.449 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:04.450 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:04.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:04.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:04.450 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:42:04.450 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:04.450 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:04.450 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:04.451 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:04.451 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:04.451 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:04.451 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:04.451 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:04.451 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:04.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:04.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:04.452 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:42:04.452 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:04.452 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:04.452 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:04.452 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:04.452 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:04.453 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:04.453 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:04.453 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:04.453 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:04.453 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:04.453 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:42:04.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:04.454 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:42:04.454 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:42:04.454 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:42:04.454 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:04.454 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:42:04.455 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T12:42:08.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:08.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:08.122 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:08.165 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:08.166 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:08.397 INFO:teuthology.orchestra.run.smithi164.stderr:No --period given, using current period=d8c95da7-75a8-4dcf-94ba-f12d5c80a46f 2020-11-09T12:42:08.432 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:42:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:08.432 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:42:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:39:43.671316Z" 2020-11-09T12:42:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925712.548101_97.1", 2020-11-09T12:42:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:41:52.548101Z" 2020-11-09T12:42:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:08.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:42:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:42:08.434 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:08.435 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:42:08.435 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T12:42:08.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:08.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:08.437 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:08.478 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:08.478 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:08.711 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:42:08.711 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:42:08.711 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:42:08.712 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:42:08.712 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:42:08.712 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:08.713 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:08.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:08.713 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:42:08.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:08.714 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:42:08.714 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:08.714 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:08.715 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:42:08.715 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:08.715 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:42:08.716 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:08.716 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:42:08.716 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:08.716 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:08.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:08.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:08.717 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:42:08.718 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:08.718 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:08.718 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925712.548101_97.1", 2020-11-09T12:42:08.718 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:08.719 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:08.719 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:08.719 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:41:52.548101Z", 2020-11-09T12:42:08.720 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:08.720 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:08.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:08.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:08.721 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:42:08.721 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:08.721 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:08.722 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:08.722 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:08.722 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:08.723 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:08.723 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:08.723 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:08.723 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:08.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:08.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:08.724 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:42:08.724 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:08.725 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:08.725 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:08.725 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:08.726 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:08.726 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:08.726 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:08.726 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:42:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:42:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:42:08.729 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:42:08.729 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:08.729 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:42:08.731 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T12:42:08.732 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T12:42:08.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:08.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:08.734 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:09.031 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:42:09.031 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:42:09.032 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:42:09.032 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:42:09.032 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:42:09.032 INFO:teuthology.orchestra.run.smithi164.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:09.032 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:09.033 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:09.033 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:42:09.033 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:09.033 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:42:09.033 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:09.033 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:09.033 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:42:09.034 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:09.034 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 1, 2020-11-09T12:42:09.034 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:09.034 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:42:09.034 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:09.034 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:09.034 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:09.035 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:09.035 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:42:09.035 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:09.035 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:09.035 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925712.548101_97.1", 2020-11-09T12:42:09.035 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:09.035 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:09.036 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:09.036 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:41:52.548101Z", 2020-11-09T12:42:09.036 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:09.036 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:09.036 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:09.036 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:09.036 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:42:09.036 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:09.037 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:09.037 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:09.037 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:09.037 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:09.037 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:09.037 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:42:09.037 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:09.038 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:09.038 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:09.038 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:09.038 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:42:09.038 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:09.038 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:42:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:42:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:42:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 1, 2020-11-09T12:42:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 1 2020-11-09T12:42:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:09.041 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:42:09.042 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T12:42:09.042 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T12:42:09.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:09.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:09.043 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:09.085 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:09.086 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:09.348 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:42:09.349 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:42:09.349 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:42:09.349 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:42:09.349 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:42:09.349 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:09.349 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:09.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:09.350 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:42:09.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:09.350 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:42:09.350 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:09.350 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:09.351 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:42:09.351 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:09.351 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:42:09.351 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:09.351 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:42:09.351 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:09.352 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:09.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:09.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:09.352 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:42:09.352 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:09.352 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:09.352 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925712.548101_97.1", 2020-11-09T12:42:09.353 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:09.353 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:09.353 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:09.353 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:41:52.548101Z", 2020-11-09T12:42:09.353 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:09.353 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:09.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:09.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:09.354 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:42:09.354 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:09.354 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:09.354 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:09.354 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:09.354 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:09.354 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:09.355 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:09.355 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:09.355 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:09.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:09.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:09.355 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:42:09.355 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:09.356 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:09.356 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:09.356 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:09.356 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:09.356 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:09.356 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:09.356 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:09.356 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:09.357 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:09.357 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:42:09.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:09.357 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:42:09.357 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:42:09.357 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:42:09.357 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:09.358 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:42:09.358 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T12:42:11.504 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-11-09T12:42:11.505 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kuljus-7 2020-11-09T12:42:12.372 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kuljus-8 2020-11-09T12:42:12.597 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kuljus-9 2020-11-09T12:42:13.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:13.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:13.525 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:13.557 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:13.558 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:13.771 INFO:teuthology.orchestra.run.smithi164.stderr:No --period given, using current period=d8c95da7-75a8-4dcf-94ba-f12d5c80a46f 2020-11-09T12:42:13.805 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:42:13.805 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:13.805 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:42:13.805 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:39:43.671316Z" 2020-11-09T12:42:13.805 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:13.806 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:13.806 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925733.502002_111.1", 2020-11-09T12:42:13.806 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:42:13.502002Z" 2020-11-09T12:42:13.806 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:13.806 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:13.806 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:13.806 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:42:13.807 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:13.807 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:13.807 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:13.807 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:42:13.807 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:13.807 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:42:13.808 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T12:42:13.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:13.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:13.810 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:13.848 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:13.849 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:14.109 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:42:14.110 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:42:14.110 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:42:14.111 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:42:14.111 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:42:14.111 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:14.111 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:14.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:14.111 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:42:14.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:14.112 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:42:14.112 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:14.112 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:14.112 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:42:14.112 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:14.112 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:42:14.112 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:14.113 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:42:14.113 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:14.113 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:14.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:14.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:14.113 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:42:14.113 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:14.114 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:14.114 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925733.502002_111.1", 2020-11-09T12:42:14.114 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:14.114 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:14.114 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:14.114 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:13.502002Z", 2020-11-09T12:42:14.114 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:14.115 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:14.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:14.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:14.115 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:42:14.115 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:14.115 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:14.115 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:14.115 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:14.116 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:14.116 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:14.116 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:14.116 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:14.116 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:14.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:14.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:14.117 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:42:14.117 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:14.117 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:14.117 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:14.117 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:14.117 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:14.117 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:14.118 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:14.118 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:14.118 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:14.118 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:14.118 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:42:14.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:14.118 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:42:14.118 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:42:14.119 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:42:14.119 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:14.119 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:42:14.120 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T12:42:14.120 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T12:42:14.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:14.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:14.122 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:14.162 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:14.162 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:14.414 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:42:14.415 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:42:14.415 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:42:14.415 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:42:14.415 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:42:14.415 INFO:teuthology.orchestra.run.smithi164.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:14.416 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:14.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:14.416 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:42:14.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:14.416 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:42:14.416 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:14.417 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:14.417 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:42:14.417 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:14.417 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 1, 2020-11-09T12:42:14.417 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:14.417 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:42:14.418 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:14.418 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:14.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:14.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:14.418 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:42:14.418 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:14.419 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:14.419 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925733.502002_111.1", 2020-11-09T12:42:14.419 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:14.419 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:14.419 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:14.419 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:42:13.502002Z", 2020-11-09T12:42:14.419 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:14.420 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:14.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:14.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:14.420 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:42:14.420 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:14.420 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:14.421 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:14.421 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:14.421 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:14.421 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:14.421 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:42:14.421 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:14.421 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:14.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:14.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:14.422 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:42:14.422 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:14.422 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:14.422 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:14.423 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:14.423 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:14.423 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:14.423 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:42:14.423 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:14.423 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:14.423 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:14.424 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:42:14.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:14.424 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:42:14.424 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 1, 2020-11-09T12:42:14.424 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 1 2020-11-09T12:42:14.424 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:14.424 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:42:14.425 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T12:42:14.426 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T12:42:14.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:14.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:14.427 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:14.467 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:14.468 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:14.737 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:42:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:42:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:42:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:42:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:42:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:42:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:14.739 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:42:14.739 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:14.739 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:14.739 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:42:14.739 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:14.739 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:42:14.739 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:14.740 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:42:14.740 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:14.740 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:14.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:14.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:14.740 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:42:14.740 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:14.740 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:14.741 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925733.502002_111.1", 2020-11-09T12:42:14.741 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:14.741 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:14.741 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:14.741 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:13.502002Z", 2020-11-09T12:42:14.741 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:14.741 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:14.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:14.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:14.742 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:42:14.742 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:14.742 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:14.742 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:14.742 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:14.743 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:14.743 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:14.743 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:14.743 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:14.743 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:14.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:14.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:14.744 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:42:14.744 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:14.744 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:14.744 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:14.744 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:14.744 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:14.744 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:14.745 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:14.745 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:14.745 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:14.745 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:14.745 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:42:14.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:14.745 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:42:14.746 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:42:14.746 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:42:14.746 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:14.746 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:42:14.747 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T12:42:16.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:16.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:16.697 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:16.734 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:16.740 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:16.933 INFO:teuthology.orchestra.run.smithi164.stderr:No --period given, using current period=d8c95da7-75a8-4dcf-94ba-f12d5c80a46f 2020-11-09T12:42:16.969 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:42:16.969 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:16.970 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:42:16.970 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:39:43.671316Z" 2020-11-09T12:42:16.970 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:16.970 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:16.970 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:42:16.970 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:42:16.653632Z" 2020-11-09T12:42:16.970 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:16.971 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:16.971 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:16.971 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:42:16.971 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:16.971 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:16.971 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:16.971 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:42:16.972 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:16.972 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:42:16.973 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T12:42:16.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:16.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:16.974 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:17.015 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:17.015 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:17.256 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:42:17.256 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:42:17.257 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:42:17.257 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:42:17.257 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:42:17.257 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:17.257 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:17.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:17.257 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:42:17.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:17.258 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:42:17.258 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:17.258 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:17.258 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:42:17.258 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:17.259 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:42:17.259 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:17.259 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:42:17.259 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:17.259 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:17.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:17.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:17.260 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:42:17.260 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:17.260 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:17.260 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:42:17.260 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:17.260 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:17.260 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:17.261 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:42:17.261 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:17.261 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:17.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:17.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:17.261 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:42:17.261 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:17.262 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:17.262 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:17.262 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:17.262 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:17.262 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:17.262 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:17.262 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:17.263 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:17.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:17.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:17.263 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:42:17.263 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:17.263 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:17.263 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:17.264 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:17.264 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:17.264 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:17.264 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:17.264 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:17.264 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:17.264 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:17.264 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:42:17.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:17.265 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:42:17.265 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:42:17.265 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:42:17.265 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:17.265 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:42:17.266 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T12:42:17.267 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T12:42:17.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:17.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:17.268 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:17.310 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:17.311 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:17.561 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:42:17.562 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:42:17.562 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:42:17.562 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:42:17.562 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:42:17.563 INFO:teuthology.orchestra.run.smithi164.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:17.563 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:17.563 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:17.563 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:42:17.563 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:17.563 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:42:17.563 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:17.564 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:17.564 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:42:17.564 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:17.564 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 1, 2020-11-09T12:42:17.564 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:17.564 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:42:17.564 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:17.565 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:17.565 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:17.565 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:17.565 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:42:17.565 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:17.565 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:17.566 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:42:17.567 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:17.567 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:17.567 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:17.567 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:42:17.567 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:17.567 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:17.567 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:17.568 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:17.568 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:42:17.568 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:17.568 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:17.568 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:17.568 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:17.569 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:17.569 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:17.569 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:42:17.569 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:17.569 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:17.569 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:17.569 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:17.570 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:42:17.570 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:17.570 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:17.570 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:17.570 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:17.570 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:17.570 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:17.571 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:42:17.571 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:17.571 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:17.571 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:17.571 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:42:17.571 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:17.571 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:42:17.571 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 1, 2020-11-09T12:42:17.572 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 1 2020-11-09T12:42:17.572 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:17.572 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:42:17.573 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T12:42:17.573 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T12:42:17.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:17.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:17.575 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:17.616 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:17.616 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:17.883 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:42:17.883 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:42:17.883 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:42:17.884 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:42:17.884 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:42:17.884 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:17.884 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:17.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:17.884 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:42:17.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:17.885 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:42:17.885 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:17.885 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:17.885 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925583.671316_42.1", 2020-11-09T12:42:17.885 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:17.885 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:42:17.885 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:17.886 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:39:43.671316Z", 2020-11-09T12:42:17.886 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:17.886 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:17.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:17.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:17.886 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:42:17.886 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:17.887 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:17.887 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:42:17.887 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:17.887 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:17.890 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:17.890 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:42:17.890 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:17.891 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:17.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:17.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:17.891 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:42:17.891 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:17.891 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:17.891 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:17.892 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:17.892 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:17.892 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:17.892 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:17.892 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:17.892 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:17.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:17.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:17.893 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:42:17.893 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:17.893 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:17.893 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:17.893 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:17.893 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:17.893 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:17.894 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:17.894 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:17.894 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:17.894 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:17.894 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:42:17.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:17.894 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:42:17.895 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:42:17.895 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:42:17.895 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:17.895 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:42:17.896 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T12:42:19.224 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-11-09T12:42:19.225 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kuljus-10 2020-11-09T12:42:19.262 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kuljus-11 2020-11-09T12:42:19.328 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kuljus-12 2020-11-09T12:42:19.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:19.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:19.988 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:20.032 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:20.032 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:20.277 INFO:teuthology.orchestra.run.smithi164.stderr:No --period given, using current period=d8c95da7-75a8-4dcf-94ba-f12d5c80a46f 2020-11-09T12:42:20.312 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:42:20.312 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:20.312 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925739.363989_54.1", 2020-11-09T12:42:20.312 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:42:19.363989Z" 2020-11-09T12:42:20.312 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:20.313 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:20.313 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:42:20.313 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:42:16.653632Z" 2020-11-09T12:42:20.313 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:20.313 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:20.313 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:20.313 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:42:20.314 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:20.314 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:20.314 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:20.314 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:42:20.314 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:20.314 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:42:20.315 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T12:42:20.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:20.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:20.316 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:20.358 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:20.358 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:20.582 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:42:20.583 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:42:20.583 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:42:20.584 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:42:20.584 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:42:20.584 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:20.585 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:20.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:20.585 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:42:20.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:20.586 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:42:20.586 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:20.587 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:20.587 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925739.363989_54.1", 2020-11-09T12:42:20.587 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:20.587 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:42:20.589 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:20.589 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:19.363989Z", 2020-11-09T12:42:20.590 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:20.590 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:20.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:20.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:20.591 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:42:20.591 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:20.591 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:20.592 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:42:20.592 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:20.592 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:20.592 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:20.592 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:42:20.593 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:20.593 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:20.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:20.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:20.593 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:42:20.594 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:20.595 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:20.595 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:20.595 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:20.596 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:20.596 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:20.596 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:20.596 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:20.596 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:20.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:20.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:20.597 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:42:20.597 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:20.597 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:20.597 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:20.597 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:20.597 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:20.597 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:20.598 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:20.598 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:20.598 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:20.598 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:20.598 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:42:20.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:20.598 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:42:20.599 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:42:20.599 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:42:20.599 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:20.599 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:42:20.600 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T12:42:20.600 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T12:42:20.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:20.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:20.602 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:20.642 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:20.642 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:20.875 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:42:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:42:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:42:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:42:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:42:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:42:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:42:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925739.363989_54.1", 2020-11-09T12:42:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 1, 2020-11-09T12:42:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:42:19.363989Z", 2020-11-09T12:42:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:42:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:42:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:42:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:42:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:42:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:42:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:42:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:42:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:42:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 1, 2020-11-09T12:42:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 1 2020-11-09T12:42:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:20.884 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:42:20.885 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T12:42:20.885 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T12:42:20.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:20.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:20.887 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:20.929 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:20.929 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:21.193 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:42:21.193 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:42:21.194 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:42:21.194 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:42:21.194 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:42:21.194 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:21.194 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:21.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:21.195 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:42:21.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:21.195 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:42:21.195 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:21.195 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:21.195 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925739.363989_54.1", 2020-11-09T12:42:21.195 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:21.196 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:42:21.196 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:21.196 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:19.363989Z", 2020-11-09T12:42:21.196 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:21.196 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:21.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:21.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:21.196 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:42:21.197 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:21.197 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:21.197 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:42:21.197 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:21.197 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:21.197 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:21.197 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:42:21.198 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:21.198 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:21.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:21.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:21.198 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:42:21.198 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:21.198 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:21.199 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:21.199 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:21.199 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:21.199 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:21.199 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:21.199 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:21.199 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:21.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:21.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:21.200 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:42:21.200 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:21.200 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:21.200 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:21.200 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:21.200 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:21.201 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:21.201 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:21.201 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:21.201 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:21.201 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:21.201 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:42:21.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:21.202 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:42:21.202 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:42:21.202 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:42:21.202 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:21.202 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:42:21.203 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T12:42:21.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:21.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:21.205 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:21.294 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:21.294 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:21.516 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:21.518+0000 7f06ec6e60c0 1 bucket sync caught up with source: 2020-11-09T12:42:21.516 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.5.5, 00000000001.7.5, , ] 2020-11-09T12:42:21.516 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.5.5, 00000000001.7.5, , ] 2020-11-09T12:42:21.517 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:21.518+0000 7f06ec6e60c0 0 bucket checkpoint complete 2020-11-09T12:42:21.547 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-10 zones={test-zone2, test-zone1} 2020-11-09T12:42:21.642 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:42:21.653 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-09T12:42:21.653 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-09T12:42:21.654 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-09T12:42:21.654 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:21.655 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:42:21.662 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-09T12:42:21.663 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-09T12:42:21.663 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-09T12:42:21.663 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:21.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:21.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:21.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:21.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:22.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:22.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:22.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:22.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:22.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:22.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:22.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:22.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:22.799 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-10 zones={test-zone2, test-zone1} 2020-11-09T12:42:22.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:22.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:22.801 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:22.848 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:22.849 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:23.062 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:23.063+0000 7f2d511cd0c0 1 bucket sync caught up with source: 2020-11-09T12:42:23.063 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.5.5, 00000000001.7.5, , ] 2020-11-09T12:42:23.063 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.5.5, 00000000001.7.5, , ] 2020-11-09T12:42:23.063 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:23.063+0000 7f2d511cd0c0 0 bucket checkpoint complete 2020-11-09T12:42:23.084 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-10 zones={test-zone3, test-zone1} 2020-11-09T12:42:24.820 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:42:24.828 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-09T12:42:24.828 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-09T12:42:24.829 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-09T12:42:24.829 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:24.829 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:42:24.833 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-09T12:42:24.834 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-09T12:42:24.834 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-09T12:42:24.835 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:24.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:24.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:25.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:25.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:25.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:25.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:25.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:25.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:25.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:25.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:25.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:25.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:25.960 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-10 zones={test-zone3, test-zone1} 2020-11-09T12:42:25.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:25.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:25.962 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:26.009 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:26.010 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:26.254 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:26.254+0000 7f67a204e0c0 1 bucket sync caught up with source: 2020-11-09T12:42:26.254 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.5.5, 00000000001.7.5, , ] 2020-11-09T12:42:26.254 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.5.5, 00000000001.7.5, , ] 2020-11-09T12:42:26.254 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:26.254+0000 7f67a204e0c0 0 bucket checkpoint complete 2020-11-09T12:42:26.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:26.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:26.382 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:26.428 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:26.429 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:26.660 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:26.663+0000 7f35a0b230c0 1 bucket sync caught up with source: 2020-11-09T12:42:26.661 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000001.6.5, , 00000000001.11.5, , , , 00000000001.7.5, 00000000001.3.5, , ] 2020-11-09T12:42:26.661 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000001.6.5, , 00000000001.11.5, , , , 00000000001.7.5, 00000000001.3.5, , ] 2020-11-09T12:42:26.661 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:26.663+0000 7f35a0b230c0 0 bucket checkpoint complete 2020-11-09T12:42:26.687 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-11 zones={test-zone1, test-zone2} 2020-11-09T12:42:27.481 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:42:27.490 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-09T12:42:27.491 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-09T12:42:27.492 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-09T12:42:27.492 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:27.492 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:42:27.499 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-09T12:42:27.500 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-09T12:42:27.500 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-09T12:42:27.501 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:27.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:27.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:27.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:27.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:27.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:27.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:28.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:28.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:28.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:28.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:28.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:28.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:28.606 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-11 zones={test-zone1, test-zone2} 2020-11-09T12:42:28.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:28.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:28.607 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:28.656 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:28.887 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:28.887+0000 7f3c448880c0 1 bucket sync caught up with source: 2020-11-09T12:42:28.887 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000001.6.5, , 00000000001.11.5, , , , 00000000001.7.5, 00000000001.3.5, , ] 2020-11-09T12:42:28.888 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000001.6.5, , 00000000001.11.5, , , , 00000000001.7.5, 00000000001.3.5, , ] 2020-11-09T12:42:28.888 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:28.887+0000 7f3c448880c0 0 bucket checkpoint complete 2020-11-09T12:42:28.915 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-11 zones={test-zone3, test-zone2} 2020-11-09T12:42:29.921 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:42:29.929 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-09T12:42:29.930 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-09T12:42:29.930 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-09T12:42:29.930 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:29.931 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:42:29.939 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-09T12:42:29.939 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-09T12:42:29.940 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-09T12:42:29.940 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:29.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:30.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:30.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:30.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:30.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:30.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:30.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:30.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:30.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:30.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:30.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:30.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:31.062 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-11 zones={test-zone3, test-zone2} 2020-11-09T12:42:31.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:31.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:31.064 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:31.108 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:31.108 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:31.348 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:31.351+0000 7f389fbfa0c0 1 bucket sync caught up with source: 2020-11-09T12:42:31.349 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000001.8.5, , 00000000001.7.5, , , , 00000000001.5.5, 00000000001.10.5, , ] 2020-11-09T12:42:31.349 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000001.8.5, , 00000000001.7.5, , , , 00000000001.5.5, 00000000001.10.5, , ] 2020-11-09T12:42:31.349 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:31.351+0000 7f389fbfa0c0 0 bucket checkpoint complete 2020-11-09T12:42:31.372 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-12 zones={test-zone1, test-zone3} 2020-11-09T12:42:31.443 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:42:31.448 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-09T12:42:31.448 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-09T12:42:31.449 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-09T12:42:31.449 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:31.449 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:42:31.454 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-09T12:42:31.454 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-09T12:42:31.455 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-09T12:42:31.455 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:31.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:31.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:31.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:31.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:31.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:31.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:31.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:32.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:32.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:32.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:32.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:32.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:32.537 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-12 zones={test-zone1, test-zone3} 2020-11-09T12:42:32.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:32.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:32.539 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:32.586 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:32.587 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:32.818 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:32.822+0000 7fb73fc6e0c0 1 bucket sync caught up with source: 2020-11-09T12:42:32.819 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000001.8.5, , 00000000001.7.5, , , , 00000000001.5.5, 00000000001.10.5, , ] 2020-11-09T12:42:32.819 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000001.8.5, , 00000000001.7.5, , , , 00000000001.5.5, 00000000001.10.5, , ] 2020-11-09T12:42:32.819 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:32.822+0000 7fb73fc6e0c0 0 bucket checkpoint complete 2020-11-09T12:42:32.849 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-12 zones={test-zone2, test-zone3} 2020-11-09T12:42:33.044 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:42:33.051 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-09T12:42:33.052 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-09T12:42:33.052 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-09T12:42:33.052 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:33.053 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:42:33.059 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-09T12:42:33.059 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-09T12:42:33.059 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-09T12:42:33.060 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:33.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:33.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:33.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-09T12:42:33.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:33.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:33.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-09T12:42:33.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:33.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:33.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-09T12:42:33.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:33.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:34.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:34.161 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-12 zones={test-zone2, test-zone3} 2020-11-09T12:42:34.162 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2020-11-09T12:42:34.163 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kuljus-13 2020-11-09T12:42:35.150 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kuljus-14 2020-11-09T12:42:35.267 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kuljus-15 2020-11-09T12:42:35.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:35.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:35.523 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:35.561 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:35.561 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:35.781 INFO:teuthology.orchestra.run.smithi164.stderr:No --period given, using current period=d8c95da7-75a8-4dcf-94ba-f12d5c80a46f 2020-11-09T12:42:35.810 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:42:35.810 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:35.810 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925755.348827_66.1", 2020-11-09T12:42:35.810 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:42:35.348827Z" 2020-11-09T12:42:35.810 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:35.811 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:35.811 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:42:35.811 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:42:16.653632Z" 2020-11-09T12:42:35.811 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:35.811 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:35.811 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:35.812 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:42:35.812 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:35.812 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:35.812 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:35.812 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:42:35.812 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:35.812 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:42:35.813 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T12:42:35.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:35.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:35.815 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:35.860 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:35.861 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:36.063 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:42:36.064 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:42:36.064 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:42:36.064 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:42:36.064 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:42:36.064 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:36.065 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:36.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:36.065 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:42:36.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:36.065 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:42:36.065 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:36.065 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:36.066 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925755.348827_66.1", 2020-11-09T12:42:36.066 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:36.066 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:42:36.066 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:36.066 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:35.348827Z", 2020-11-09T12:42:36.066 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:36.066 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:36.067 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:36.067 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:36.067 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:42:36.067 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:36.067 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:36.067 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:42:36.067 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:36.068 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:36.068 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:36.068 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:42:36.068 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:36.068 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:36.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:36.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:36.069 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:42:36.069 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:36.069 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:36.069 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:36.069 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:36.070 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:36.070 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:36.070 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:36.070 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:36.070 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:36.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:36.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:36.071 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:42:36.071 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:36.071 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:36.071 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:36.071 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:36.071 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:36.071 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:36.072 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:36.072 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:36.072 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:36.072 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:36.072 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:42:36.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:36.072 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:42:36.073 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:42:36.073 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:42:36.073 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:36.073 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:42:36.074 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T12:42:36.074 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T12:42:36.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:36.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:36.076 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:36.115 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:36.116 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:36.345 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:42:36.346 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:42:36.346 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:42:36.346 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:42:36.346 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:42:36.346 INFO:teuthology.orchestra.run.smithi164.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:36.346 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:36.347 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:36.347 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:42:36.347 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:36.347 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:42:36.347 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:36.347 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:36.347 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925755.348827_66.1", 2020-11-09T12:42:36.348 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:36.348 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 1, 2020-11-09T12:42:36.348 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:36.348 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:42:35.348827Z", 2020-11-09T12:42:36.348 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:36.348 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:36.348 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:36.349 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:36.349 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:42:36.349 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:36.349 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:36.349 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:42:36.349 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:36.349 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:36.350 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:36.350 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:42:36.350 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:36.350 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:36.350 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:36.350 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:36.350 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:42:36.351 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:36.351 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:36.351 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:36.351 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:36.351 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:36.351 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:36.351 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:42:36.352 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:36.352 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:36.352 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:36.352 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:42:36.352 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:42:36.352 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:42:36.352 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:42:36.353 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:42:36.353 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:42:36.354 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:42:36.354 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:42:36.354 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:42:36.354 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:42:36.354 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:36.354 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:36.354 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:42:36.355 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:42:36.355 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:42:36.355 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 1, 2020-11-09T12:42:36.355 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 1 2020-11-09T12:42:36.355 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:42:36.355 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:42:36.356 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T12:42:36.357 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T12:42:36.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:36.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:42:36.358 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:42:36.399 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:36.399 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:36.763 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:42:36.763 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:42:36.763 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:42:36.763 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:42:36.764 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:42:36.764 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:42:36.764 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:36.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:36.764 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:42:36.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:36.764 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:42:36.765 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:36.765 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:36.765 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925755.348827_66.1", 2020-11-09T12:42:36.765 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:36.765 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:42:36.765 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:36.766 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:35.348827Z", 2020-11-09T12:42:36.766 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:36.766 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:36.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:36.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:36.766 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:42:36.766 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:36.767 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:36.767 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:42:36.767 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:36.767 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:36.767 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:36.767 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:42:36.767 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:36.767 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:36.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:36.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:36.768 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:42:36.768 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:36.768 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:36.768 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:36.768 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:36.769 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:36.769 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:36.769 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:36.769 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:36.769 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:36.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:36.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:42:36.770 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:42:36.770 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:42:36.770 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:42:36.770 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:42:36.770 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:42:36.771 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:42:36.771 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:42:36.771 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:42:36.771 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:42:36.771 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:36.771 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:36.771 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:42:36.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:42:36.772 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:42:36.772 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:42:36.772 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:42:36.772 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:42:36.772 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:42:36.773 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T12:42:36.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:36.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:36.775 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:36.869 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:36.869 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:37.099 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:37.102+0000 7fac19cc40c0 1 bucket sync caught up with source: 2020-11-09T12:42:37.099 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2020-11-09T12:42:37.100 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000001.11.5, , , , , , , , , ] 2020-11-09T12:42:37.100 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:37.102+0000 7fac19cc40c0 0 bucket checkpoint complete 2020-11-09T12:42:37.130 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-13 zones={test-zone2, test-zone1} 2020-11-09T12:42:38.385 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:42:38.392 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:38.393 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:42:38.442 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:38.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:38.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:38.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:38.777 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-13 zones={test-zone2, test-zone1} 2020-11-09T12:42:38.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:38.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:38.779 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:38.822 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:38.822 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:39.044 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:39.043+0000 7fc2ee9820c0 1 bucket sync caught up with source: 2020-11-09T12:42:39.044 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2020-11-09T12:42:39.044 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000001.11.5, , , , , , , , , ] 2020-11-09T12:42:39.045 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:39.043+0000 7fc2ee9820c0 0 bucket checkpoint complete 2020-11-09T12:42:39.072 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-13 zones={test-zone3, test-zone1} 2020-11-09T12:42:39.383 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:42:39.430 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:39.431 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:42:39.437 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:39.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:39.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:39.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:39.770 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-13 zones={test-zone3, test-zone1} 2020-11-09T12:42:39.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:39.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:39.772 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:39.813 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:39.814 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:40.036 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:40.038+0000 7f95dc4510c0 1 bucket sync caught up with source: 2020-11-09T12:42:40.037 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2020-11-09T12:42:40.037 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000001.11.5, , , , , , , , , ] 2020-11-09T12:42:40.037 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:40.038+0000 7f95dc4510c0 0 bucket checkpoint complete 2020-11-09T12:42:40.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:40.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:40.079 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:40.130 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:40.131 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:40.351 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:40.351+0000 7febcd7770c0 1 bucket sync caught up with source: 2020-11-09T12:42:40.352 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-09T12:42:40.352 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2020-11-09T12:42:40.352 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:40.351+0000 7febcd7770c0 0 bucket checkpoint complete 2020-11-09T12:42:40.381 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-14 zones={test-zone1, test-zone2} 2020-11-09T12:42:41.058 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:42:41.066 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:41.066 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:42:41.114 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:41.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:41.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:41.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:41.453 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-14 zones={test-zone1, test-zone2} 2020-11-09T12:42:41.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:41.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:41.455 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:41.723 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:41.723+0000 7fa4c3d580c0 1 bucket sync caught up with source: 2020-11-09T12:42:41.724 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-09T12:42:41.724 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2020-11-09T12:42:41.724 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:41.723+0000 7fa4c3d580c0 0 bucket checkpoint complete 2020-11-09T12:42:41.751 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-14 zones={test-zone3, test-zone2} 2020-11-09T12:42:41.837 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:42:41.886 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:41.887 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:42:41.894 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:41.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:42.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:42.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:42.266 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-14 zones={test-zone3, test-zone2} 2020-11-09T12:42:42.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:42.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:42.268 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:42.315 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:42.316 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:42.546 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:42.547+0000 7fb573a600c0 1 bucket sync caught up with source: 2020-11-09T12:42:42.546 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000001.7.5, , , , , , , , , ] 2020-11-09T12:42:42.546 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000001.7.5, , , , , , , , , ] 2020-11-09T12:42:42.546 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:42.547+0000 7fb573a600c0 0 bucket checkpoint complete 2020-11-09T12:42:42.572 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-15 zones={test-zone1, test-zone3} 2020-11-09T12:42:42.751 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:42:42.798 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:42.799 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:42:42.806 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:42.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:42.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:43.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:43.129 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-15 zones={test-zone1, test-zone3} 2020-11-09T12:42:43.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:43.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:43.131 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:43.178 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:43.179 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:43.417 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:43.418+0000 7efd9c2240c0 1 bucket sync caught up with source: 2020-11-09T12:42:43.417 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000001.7.5, , , , , , , , , ] 2020-11-09T12:42:43.418 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000001.7.5, , , , , , , , , ] 2020-11-09T12:42:43.418 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:43.418+0000 7efd9c2240c0 0 bucket checkpoint complete 2020-11-09T12:42:43.445 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-15 zones={test-zone2, test-zone3} 2020-11-09T12:42:44.453 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:42:44.462 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:44.463 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:42:44.514 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-09T12:42:44.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:44.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:44.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-09T12:42:44.841 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-15 zones={test-zone2, test-zone3} 2020-11-09T12:42:45.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:45.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:45.366 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:45.413 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:45.414 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:45.640 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:45.642+0000 7f24935040c0 1 waiting for incremental sync to catch up: 2020-11-09T12:42:45.640 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2020-11-09T12:42:45.640 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2020-11-09T12:42:50.644 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:50.646+0000 7f24935040c0 1 waiting for incremental sync to catch up: 2020-11-09T12:42:50.644 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2020-11-09T12:42:50.644 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2020-11-09T12:42:55.646 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:55.650+0000 7f24935040c0 1 bucket sync caught up with source: 2020-11-09T12:42:55.647 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000002.13.5, , , , , , , , , ] 2020-11-09T12:42:55.647 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2020-11-09T12:42:55.647 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:55.650+0000 7f24935040c0 0 bucket checkpoint complete 2020-11-09T12:42:55.674 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-13 zones={test-zone2, test-zone1} 2020-11-09T12:42:56.393 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:42:56.402 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:42:56.506 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-13 zones={test-zone2, test-zone1} 2020-11-09T12:42:56.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:56.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:56.508 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:56.556 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:56.557 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:56.769 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:56.771+0000 7fa6d9e4e0c0 1 bucket sync caught up with source: 2020-11-09T12:42:56.769 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000002.13.5, , , , , , , , , ] 2020-11-09T12:42:56.769 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2020-11-09T12:42:56.769 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:56.771+0000 7fa6d9e4e0c0 0 bucket checkpoint complete 2020-11-09T12:42:56.795 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-13 zones={test-zone3, test-zone1} 2020-11-09T12:42:57.203 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:42:57.254 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:42:57.316 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-13 zones={test-zone3, test-zone1} 2020-11-09T12:42:57.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:57.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:57.318 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:57.366 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:57.367 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:57.605 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:57.610+0000 7fc3ed2190c0 1 bucket sync caught up with source: 2020-11-09T12:42:57.606 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000002.13.5, , , , , , , , , ] 2020-11-09T12:42:57.606 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2020-11-09T12:42:57.606 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:42:57.610+0000 7fc3ed2190c0 0 bucket checkpoint complete 2020-11-09T12:42:58.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:58.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:42:58.230 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:42:58.277 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:42:58.277 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:42:58.503 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:42:58.503+0000 7fca91a420c0 1 waiting for incremental sync to catch up: 2020-11-09T12:42:58.503 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-09T12:42:58.503 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-11-09T12:43:03.506 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:43:03.507+0000 7fca91a420c0 1 waiting for incremental sync to catch up: 2020-11-09T12:43:03.507 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-09T12:43:03.507 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-11-09T12:43:08.510 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:43:08.511+0000 7fca91a420c0 1 waiting for incremental sync to catch up: 2020-11-09T12:43:08.510 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-09T12:43:08.511 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-11-09T12:43:13.514 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:43:13.515+0000 7fca91a420c0 1 waiting for incremental sync to catch up: 2020-11-09T12:43:13.515 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-09T12:43:13.515 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-11-09T12:43:18.518 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:43:18.519+0000 7fca91a420c0 1 bucket sync caught up with source: 2020-11-09T12:43:18.518 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2020-11-09T12:43:18.519 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-11-09T12:43:18.519 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:43:18.519+0000 7fca91a420c0 0 bucket checkpoint complete 2020-11-09T12:43:18.551 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-14 zones={test-zone1, test-zone2} 2020-11-09T12:43:19.086 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:43:19.092 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:43:19.199 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-14 zones={test-zone1, test-zone2} 2020-11-09T12:43:19.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:43:19.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:43:19.200 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:43:19.249 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:43:19.249 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:43:19.469 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:43:19.471+0000 7f15ae95d0c0 1 bucket sync caught up with source: 2020-11-09T12:43:19.470 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2020-11-09T12:43:19.470 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-11-09T12:43:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:43:19.471+0000 7f15ae95d0c0 0 bucket checkpoint complete 2020-11-09T12:43:19.504 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-14 zones={test-zone3, test-zone2} 2020-11-09T12:43:19.966 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:43:20.014 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:43:20.077 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-14 zones={test-zone3, test-zone2} 2020-11-09T12:43:20.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:43:20.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:43:20.100 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:43:20.141 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:43:20.142 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:43:20.360 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:43:20.363+0000 7f86ca6e20c0 1 bucket sync caught up with source: 2020-11-09T12:43:20.360 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000002.13.5, , , , , , , , , ] 2020-11-09T12:43:20.360 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2020-11-09T12:43:20.361 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:43:20.363+0000 7f86ca6e20c0 0 bucket checkpoint complete 2020-11-09T12:43:20.388 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-15 zones={test-zone1, test-zone3} 2020-11-09T12:43:20.941 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:43:20.993 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:43:21.076 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-15 zones={test-zone1, test-zone3} 2020-11-09T12:43:21.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:43:21.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:43:21.077 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:43:21.126 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:43:21.126 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:43:21.349 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:43:21.354+0000 7f18850370c0 1 bucket sync caught up with source: 2020-11-09T12:43:21.350 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000002.13.5, , , , , , , , , ] 2020-11-09T12:43:21.350 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2020-11-09T12:43:21.350 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:43:21.354+0000 7f18850370c0 0 bucket checkpoint complete 2020-11-09T12:43:21.379 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-15 zones={test-zone2, test-zone3} 2020-11-09T12:43:21.579 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:43:21.634 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:43:21.704 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-15 zones={test-zone2, test-zone3} 2020-11-09T12:43:21.705 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2020-11-09T12:43:21.705 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kuljus-16 2020-11-09T12:43:21.737 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kuljus-17 2020-11-09T12:43:21.801 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kuljus-18 2020-11-09T12:43:21.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:43:21.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:43:21.910 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:43:21.945 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:43:21.946 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:43:22.163 INFO:teuthology.orchestra.run.smithi164.stderr:No --period given, using current period=d8c95da7-75a8-4dcf-94ba-f12d5c80a46f 2020-11-09T12:43:22.193 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:43:22.194 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:43:22.194 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925801.904842_88.1", 2020-11-09T12:43:22.194 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:43:21.904842Z" 2020-11-09T12:43:22.195 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:43:22.195 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:43:22.195 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:43:22.196 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:42:16.653632Z" 2020-11-09T12:43:22.196 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:43:22.196 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:43:22.199 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:43:22.199 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:43:22.199 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:43:22.200 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:43:22.200 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:43:22.200 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:43:22.201 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:43:22.201 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:43:22.202 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T12:43:22.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:43:22.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:43:22.205 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:43:22.247 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:43:22.248 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:43:22.487 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:43:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:43:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:43:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:43:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:43:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:43:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:43:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:43:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:43:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:43:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:43:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:43:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:43:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925801.904842_88.1", 2020-11-09T12:43:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:43:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:43:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:43:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:43:21.904842Z", 2020-11-09T12:43:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:43:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:43:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:43:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:43:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:43:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:43:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:43:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:43:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:43:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:43:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:43:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:43:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:43:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:43:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:43:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:43:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:43:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:43:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:43:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:43:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:43:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:43:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:43:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:43:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:43:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:43:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:43:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:43:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:43:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:43:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:43:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:43:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:43:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:43:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:43:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:43:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:43:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:43:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:43:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:43:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:43:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:43:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:43:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:43:22.496 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:43:22.496 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:43:22.497 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T12:43:22.497 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T12:43:22.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:43:22.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:43:22.498 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:43:22.540 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:43:22.540 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:43:22.779 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:43:22.780 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:43:22.780 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:43:22.780 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:43:22.780 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:43:22.780 INFO:teuthology.orchestra.run.smithi164.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:43:22.781 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:43:22.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:43:22.781 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:43:22.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:43:22.781 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:43:22.781 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:43:22.781 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:43:22.782 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925801.904842_88.1", 2020-11-09T12:43:22.782 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:43:22.782 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 1, 2020-11-09T12:43:22.782 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:43:22.782 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:43:21.904842Z", 2020-11-09T12:43:22.782 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:43:22.782 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:43:22.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:43:22.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:43:22.783 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:43:22.783 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:43:22.783 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:43:22.783 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:43:22.783 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:43:22.784 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:43:22.784 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:43:22.784 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:43:22.784 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:43:22.784 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:43:22.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:43:22.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:43:22.785 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:43:22.785 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:43:22.785 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:43:22.785 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:43:22.785 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:43:22.785 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:43:22.785 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:43:22.786 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:43:22.786 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:43:22.786 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:43:22.786 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:43:22.786 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:43:22.786 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:43:22.786 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:43:22.787 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:43:22.787 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:43:22.787 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:43:22.787 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:43:22.787 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:43:22.787 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:43:22.787 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:43:22.788 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:43:22.788 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:43:22.788 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:43:22.788 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:43:22.788 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:43:22.788 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 1, 2020-11-09T12:43:22.788 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 1 2020-11-09T12:43:22.789 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:43:22.789 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:43:22.790 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T12:43:22.790 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T12:43:22.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:43:22.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:43:22.791 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:43:22.831 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:43:22.831 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:43:23.294 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:43:23.295 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:43:23.295 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:43:23.295 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:43:23.295 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:43:23.295 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:43:23.296 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:43:23.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:43:23.296 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:43:23.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:43:23.296 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:43:23.296 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:43:23.296 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:43:23.297 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925801.904842_88.1", 2020-11-09T12:43:23.297 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:43:23.297 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:43:23.297 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:43:23.297 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:43:21.904842Z", 2020-11-09T12:43:23.297 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:43:23.297 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:43:23.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:43:23.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:43:23.298 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:43:23.298 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:43:23.298 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:43:23.298 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:43:23.299 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:43:23.299 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:43:23.299 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:43:23.299 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:43:23.299 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:43:23.299 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:43:23.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:43:23.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:43:23.300 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:43:23.300 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:43:23.300 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:43:23.300 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:43:23.300 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:43:23.300 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:43:23.301 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:43:23.301 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:43:23.301 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:43:23.301 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:43:23.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:43:23.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:43:23.301 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:43:23.302 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:43:23.302 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:43:23.302 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:43:23.302 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:43:23.302 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:43:23.302 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:43:23.302 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:43:23.303 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:43:23.303 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:43:23.303 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:43:23.303 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:43:23.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:43:23.303 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:43:23.303 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:43:23.304 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:43:23.304 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:43:23.305 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:43:23.306 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T12:43:24.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:43:24.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:43:24.056 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:43:24.104 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:43:24.104 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:43:24.329 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:43:24.334+0000 7fd99297c0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:43:24.329 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , , , , , , ] 2020-11-09T12:43:24.330 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000002.12.10, , , , , ] 2020-11-09T12:43:29.333 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:43:29.338+0000 7fd99297c0c0 1 bucket sync caught up with source: 2020-11-09T12:43:29.333 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000002.12.10, , , , , ] 2020-11-09T12:43:29.333 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000002.12.10, , , , , ] 2020-11-09T12:43:29.333 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:43:29.338+0000 7fd99297c0c0 0 bucket checkpoint complete 2020-11-09T12:43:29.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:43:29.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:43:29.367 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:43:29.415 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:43:29.415 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:43:29.650 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:43:29.651+0000 7fadcd1270c0 1 waiting for incremental sync to catch up: 2020-11-09T12:43:29.651 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000002.12.10, , , , , ] 2020-11-09T12:43:29.651 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-09T12:43:34.655 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:43:34.655+0000 7fadcd1270c0 1 waiting for incremental sync to catch up: 2020-11-09T12:43:34.655 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000002.12.10, , , , , ] 2020-11-09T12:43:34.655 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-09T12:43:39.659 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:43:39.663+0000 7fadcd1270c0 1 waiting for incremental sync to catch up: 2020-11-09T12:43:39.659 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000002.12.10, , , , , ] 2020-11-09T12:43:39.660 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-09T12:43:44.664 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:43:44.667+0000 7fadcd1270c0 1 bucket sync caught up with source: 2020-11-09T12:43:44.664 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000005.18.9, , , , , ] 2020-11-09T12:43:44.664 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.16.9, , , , , ] 2020-11-09T12:43:44.664 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:43:44.667+0000 7fadcd1270c0 0 bucket checkpoint complete 2020-11-09T12:43:44.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:43:44.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:43:44.696 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:43:44.743 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:43:44.744 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:43:44.984 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:43:44.990+0000 7fa04fc090c0 1 waiting for incremental sync to catch up: 2020-11-09T12:43:44.984 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000004.16.9, , , , , ] 2020-11-09T12:43:44.984 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.18.9, , , , , ] 2020-11-09T12:43:49.988 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:43:49.994+0000 7fa04fc090c0 1 waiting for incremental sync to catch up: 2020-11-09T12:43:49.988 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000004.16.9, , , , , ] 2020-11-09T12:43:49.988 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.18.9, , , , , ] 2020-11-09T12:43:54.992 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:43:54.998+0000 7fa04fc090c0 1 waiting for incremental sync to catch up: 2020-11-09T12:43:54.992 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000004.16.9, , , , , ] 2020-11-09T12:43:54.992 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.18.9, , , , , ] 2020-11-09T12:43:59.997 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:44:00.001+0000 7fa04fc090c0 1 waiting for incremental sync to catch up: 2020-11-09T12:43:59.997 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000004.16.9, , , , , ] 2020-11-09T12:43:59.997 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.18.9, , , , , ] 2020-11-09T12:44:05.002 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:44:05.008+0000 7fa04fc090c0 1 bucket sync caught up with source: 2020-11-09T12:44:05.003 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000005.18.9, , , , , ] 2020-11-09T12:44:05.003 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.18.9, , , , , ] 2020-11-09T12:44:05.003 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:44:05.008+0000 7fa04fc090c0 0 bucket checkpoint complete 2020-11-09T12:44:06.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:44:06.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:44:06.040 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:44:06.088 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:44:06.089 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:44:06.310 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:44:06.309+0000 7ff7ced570c0 1 bucket sync caught up with source: 2020-11-09T12:44:06.310 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000002.18.10, , , , , ] 2020-11-09T12:44:06.310 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000002.18.10, , , , , ] 2020-11-09T12:44:06.311 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:44:06.309+0000 7ff7ced570c0 0 bucket checkpoint complete 2020-11-09T12:44:06.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:44:06.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:44:06.334 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:44:06.377 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:44:06.378 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:44:06.618 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:44:06.617+0000 7f828f8fc0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:44:06.618 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000002.18.10, , , , , ] 2020-11-09T12:44:06.618 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.22.9, , , , , ] 2020-11-09T12:44:11.622 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:44:11.621+0000 7f828f8fc0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:44:11.622 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000002.18.10, , , , , ] 2020-11-09T12:44:11.623 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.22.9, , , , , ] 2020-11-09T12:44:16.626 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:44:16.624+0000 7f828f8fc0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:44:16.626 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000002.18.10, , , , , ] 2020-11-09T12:44:16.627 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.22.9, , , , , ] 2020-11-09T12:44:21.630 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:44:21.628+0000 7f828f8fc0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:44:21.630 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000002.18.10, , , , , ] 2020-11-09T12:44:21.631 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.22.9, , , , , ] 2020-11-09T12:44:26.634 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:44:26.635+0000 7f828f8fc0c0 1 bucket sync caught up with source: 2020-11-09T12:44:26.634 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000005.24.9, , , , , ] 2020-11-09T12:44:26.635 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.22.9, , , , , ] 2020-11-09T12:44:26.635 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:44:26.635+0000 7f828f8fc0c0 0 bucket checkpoint complete 2020-11-09T12:44:26.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:44:26.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:44:26.716 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:44:26.764 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:44:26.764 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:44:26.976 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:44:26.975+0000 7f23b7bc50c0 1 waiting for incremental sync to catch up: 2020-11-09T12:44:26.977 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , , , , , , ] 2020-11-09T12:44:26.977 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000002.16.10, , , , , ] 2020-11-09T12:44:31.981 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:44:31.979+0000 7f23b7bc50c0 1 bucket sync caught up with source: 2020-11-09T12:44:31.981 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000002.16.10, , , , , ] 2020-11-09T12:44:31.981 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000002.16.10, , , , , ] 2020-11-09T12:44:31.982 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:44:31.979+0000 7f23b7bc50c0 0 bucket checkpoint complete 2020-11-09T12:44:32.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:44:32.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:44:32.011 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:44:32.059 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:44:32.059 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:44:32.283 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:44:32.284+0000 7eff526ff0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:44:32.284 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000002.16.10, , , , , ] 2020-11-09T12:44:32.284 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000004.20.9, , , , , ] 2020-11-09T12:44:37.288 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:44:37.288+0000 7eff526ff0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:44:37.289 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000002.16.10, , , , , ] 2020-11-09T12:44:37.289 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000004.20.9, , , , , ] 2020-11-09T12:44:42.293 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:44:42.291+0000 7eff526ff0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:44:42.293 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000002.16.10, , , , , ] 2020-11-09T12:44:42.294 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000004.20.9, , , , , ] 2020-11-09T12:44:47.297 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:44:47.295+0000 7eff526ff0c0 1 bucket sync caught up with source: 2020-11-09T12:44:47.297 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000004.20.9, , , , , ] 2020-11-09T12:44:47.297 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000004.20.9, , , , , ] 2020-11-09T12:44:47.297 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:44:47.295+0000 7eff526ff0c0 0 bucket checkpoint complete 2020-11-09T12:44:50.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:44:50.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:44:50.738 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:44:50.790 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:44:50.791 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:44:51.008 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:44:51.006+0000 7fdd9cd600c0 1 waiting for incremental sync to catch up: 2020-11-09T12:44:51.008 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000010.26.5, , 00000000002.9.10, , 00000000008.24.9, , , , , ] 2020-11-09T12:44:51.009 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000010.26.5, , 00000000003.11.9, , 00000000008.24.9, , , , , ] 2020-11-09T12:44:56.012 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:44:56.010+0000 7fdd9cd600c0 1 waiting for incremental sync to catch up: 2020-11-09T12:44:56.013 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000010.26.5, , 00000000002.9.10, , 00000000008.24.9, , , , , ] 2020-11-09T12:44:56.013 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000010.26.5, , 00000000003.11.9, , 00000000008.24.9, , , , , ] 2020-11-09T12:45:01.017 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:01.014+0000 7fdd9cd600c0 1 waiting for incremental sync to catch up: 2020-11-09T12:45:01.017 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000010.26.5, , 00000000002.9.10, , 00000000008.24.9, , , , , ] 2020-11-09T12:45:01.017 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000010.26.5, , 00000000003.11.9, , 00000000008.24.9, , , , , ] 2020-11-09T12:45:06.021 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:06.021+0000 7fdd9cd600c0 1 waiting for incremental sync to catch up: 2020-11-09T12:45:06.022 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000010.26.5, , 00000000002.9.10, , 00000000008.24.9, , , , , ] 2020-11-09T12:45:06.022 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000010.26.5, , 00000000003.11.9, , 00000000008.24.9, , , , , ] 2020-11-09T12:45:11.026 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:11.024+0000 7fdd9cd600c0 1 bucket sync caught up with source: 2020-11-09T12:45:11.026 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000010.26.5, , 00000000003.11.9, , 00000000008.24.9, , , , , ] 2020-11-09T12:45:11.027 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000010.26.5, , 00000000003.11.9, , 00000000008.24.9, , , , , ] 2020-11-09T12:45:11.027 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:11.024+0000 7fdd9cd600c0 0 bucket checkpoint complete 2020-11-09T12:45:11.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:11.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:11.055 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:45:11.100 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:45:11.100 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:45:11.327 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:11.325+0000 7f773e3380c0 1 bucket sync caught up with source: 2020-11-09T12:45:11.328 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000010.26.5, , 00000000003.11.9, , 00000000008.24.9, , , , , ] 2020-11-09T12:45:11.328 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000010.26.5, , 00000000003.11.9, , 00000000008.24.9, , , , , ] 2020-11-09T12:45:11.328 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:11.325+0000 7f773e3380c0 0 bucket checkpoint complete 2020-11-09T12:45:11.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:11.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:11.353 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:45:11.401 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:45:11.401 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:45:11.641 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:11.640+0000 7fd858ffc0c0 1 bucket sync caught up with source: 2020-11-09T12:45:11.641 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000010.26.5, , 00000000003.11.9, , 00000000008.24.9, , , , , ] 2020-11-09T12:45:11.642 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000010.26.5, , 00000000003.11.9, , 00000000008.24.9, , , , , ] 2020-11-09T12:45:11.642 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:11.640+0000 7fd858ffc0c0 0 bucket checkpoint complete 2020-11-09T12:45:11.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:11.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:11.770 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:45:11.817 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:45:11.818 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:45:12.033 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:12.029+0000 7f46feed00c0 1 bucket sync caught up with source: 2020-11-09T12:45:12.033 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000003.13.9, , 00000000010.28.5, , 00000000007.29.9, , , , , ] 2020-11-09T12:45:12.034 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000003.13.9, , 00000000010.28.5, , 00000000007.29.9, , , , , ] 2020-11-09T12:45:12.034 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:12.029+0000 7f46feed00c0 0 bucket checkpoint complete 2020-11-09T12:45:12.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:12.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:12.065 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:45:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:45:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:45:12.335 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:12.329+0000 7facbec910c0 1 bucket sync caught up with source: 2020-11-09T12:45:12.335 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000003.13.9, , 00000000010.28.5, , 00000000007.29.9, , , , , ] 2020-11-09T12:45:12.336 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000003.13.9, , 00000000010.28.5, , 00000000007.29.9, , , , , ] 2020-11-09T12:45:12.336 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:12.329+0000 7facbec910c0 0 bucket checkpoint complete 2020-11-09T12:45:12.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:12.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:12.366 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:45:12.452 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:45:12.453 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:45:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:12.681+0000 7f468a4ad0c0 1 bucket sync caught up with source: 2020-11-09T12:45:12.683 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000003.14.9, , 00000000003.17.9, , 00000000014.41.5, , , , , ] 2020-11-09T12:45:12.684 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000003.14.9, , 00000000003.17.9, , 00000000014.41.5, , , , , ] 2020-11-09T12:45:12.684 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:12.681+0000 7f468a4ad0c0 0 bucket checkpoint complete 2020-11-09T12:45:12.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:12.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:12.710 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:45:12.758 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:45:12.758 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:45:12.991 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:12.988+0000 7f757e29c0c0 1 bucket sync caught up with source: 2020-11-09T12:45:12.991 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000003.14.9, , 00000000003.17.9, , 00000000014.41.5, , , , , ] 2020-11-09T12:45:12.991 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000003.14.9, , 00000000003.17.9, , 00000000014.41.5, , , , , ] 2020-11-09T12:45:12.991 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:12.988+0000 7f757e29c0c0 0 bucket checkpoint complete 2020-11-09T12:45:13.028 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-16 zones={test-zone2, test-zone1} 2020-11-09T12:45:14.556 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:45:14.567 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:45:14.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:45:14.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:45:14.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:45:14.570 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:45:14.580 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:45:14.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:45:14.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:45:14.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:45:14.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:14.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:14.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:14.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:14.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:14.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:15.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:15.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:15.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:15.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:15.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:15.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:15.610 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-16 zones={test-zone2, test-zone1} 2020-11-09T12:45:15.611 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-16 zones={test-zone3, test-zone1} 2020-11-09T12:45:16.023 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:45:16.028 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:45:16.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:45:16.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:45:16.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:45:16.029 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:45:16.034 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:45:16.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:45:16.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:45:16.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:45:16.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:16.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:16.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:16.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:16.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:16.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:16.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:16.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:16.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:16.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:16.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:16.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:17.001 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-16 zones={test-zone3, test-zone1} 2020-11-09T12:45:17.002 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-16 zones={test-zone3, test-zone2} 2020-11-09T12:45:17.824 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:45:17.832 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:45:17.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:45:17.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:45:17.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:45:17.833 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:45:17.840 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:45:17.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:45:17.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:45:17.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:45:17.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:17.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:17.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:17.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:18.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:18.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:18.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:18.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:18.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:18.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:18.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:18.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:18.828 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-16 zones={test-zone3, test-zone2} 2020-11-09T12:45:18.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:18.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:18.829 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:45:18.874 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:45:18.874 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:45:19.113 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:19.107+0000 7f337974e0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:45:19.113 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000010.29.5, , 00000000003.38.9, , 00000000004.16.9, , , , , ] 2020-11-09T12:45:19.113 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000010.29.5, , 00000000003.38.9, , 00000000007.22.9, , , , , ] 2020-11-09T12:45:24.117 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:24.114+0000 7f337974e0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:45:24.118 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000010.29.5, , 00000000003.38.9, , 00000000004.16.9, , , , , ] 2020-11-09T12:45:24.118 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000010.29.5, , 00000000003.38.9, , 00000000007.22.9, , , , , ] 2020-11-09T12:45:29.123 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:29.117+0000 7f337974e0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:45:29.123 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000010.29.5, , 00000000003.38.9, , 00000000004.16.9, , , , , ] 2020-11-09T12:45:29.124 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000010.29.5, , 00000000003.38.9, , 00000000007.22.9, , , , , ] 2020-11-09T12:45:34.127 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:34.125+0000 7f337974e0c0 1 bucket sync caught up with source: 2020-11-09T12:45:34.128 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000010.29.5, , 00000000003.38.9, , 00000000007.22.9, , , , , ] 2020-11-09T12:45:34.128 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000010.29.5, , 00000000003.38.9, , 00000000007.22.9, , , , , ] 2020-11-09T12:45:34.128 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:34.125+0000 7f337974e0c0 0 bucket checkpoint complete 2020-11-09T12:45:34.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:34.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:34.160 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:45:34.204 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:45:34.205 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:45:34.414 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:34.408+0000 7fd870e130c0 1 bucket sync caught up with source: 2020-11-09T12:45:34.415 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000010.29.5, , 00000000003.38.9, , 00000000007.22.9, , , , , ] 2020-11-09T12:45:34.415 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000010.29.5, , 00000000003.38.9, , 00000000007.22.9, , , , , ] 2020-11-09T12:45:34.415 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:34.408+0000 7fd870e130c0 0 bucket checkpoint complete 2020-11-09T12:45:34.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:34.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:34.441 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:45:34.487 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:45:34.488 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:45:34.725 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:34.721+0000 7ff1846530c0 1 bucket sync caught up with source: 2020-11-09T12:45:34.725 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000010.29.5, , 00000000003.38.9, , 00000000007.22.9, , , , , ] 2020-11-09T12:45:34.725 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000010.29.5, , 00000000003.38.9, , 00000000007.22.9, , , , , ] 2020-11-09T12:45:34.726 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:34.721+0000 7ff1846530c0 0 bucket checkpoint complete 2020-11-09T12:45:34.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:34.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:34.852 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:45:34.899 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:45:34.899 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:45:35.118 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:35.112+0000 7f10d454a0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:45:35.118 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000003.23.9, , 00000000010.43.5, , 00000000008.30.10, , , , , ] 2020-11-09T12:45:35.118 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000003.23.9, , 00000000010.43.5, , 00000000009.32.9, , , , , ] 2020-11-09T12:45:40.122 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:40.116+0000 7f10d454a0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:45:40.122 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000003.23.9, , 00000000010.43.5, , 00000000008.30.10, , , , , ] 2020-11-09T12:45:40.123 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000003.23.9, , 00000000010.43.5, , 00000000009.32.9, , , , , ] 2020-11-09T12:45:45.126 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:45.120+0000 7f10d454a0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:45:45.126 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000003.23.9, , 00000000010.43.5, , 00000000008.30.10, , , , , ] 2020-11-09T12:45:45.126 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000003.23.9, , 00000000010.43.5, , 00000000009.32.9, , , , , ] 2020-11-09T12:45:50.129 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:50.124+0000 7f10d454a0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:45:50.130 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000003.23.9, , 00000000010.43.5, , 00000000008.30.10, , , , , ] 2020-11-09T12:45:50.130 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000003.23.9, , 00000000010.43.5, , 00000000009.32.9, , , , , ] 2020-11-09T12:45:55.133 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:55.128+0000 7f10d454a0c0 1 bucket sync caught up with source: 2020-11-09T12:45:55.134 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000003.23.9, , 00000000010.43.5, , 00000000009.32.9, , , , , ] 2020-11-09T12:45:55.134 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000003.23.9, , 00000000010.43.5, , 00000000009.32.9, , , , , ] 2020-11-09T12:45:55.134 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:55.128+0000 7f10d454a0c0 0 bucket checkpoint complete 2020-11-09T12:45:55.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:55.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:55.169 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:45:55.215 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:45:55.215 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:45:55.446 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:55.440+0000 7faabe3f10c0 1 bucket sync caught up with source: 2020-11-09T12:45:55.446 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000003.23.9, , 00000000010.43.5, , 00000000009.32.9, , , , , ] 2020-11-09T12:45:55.446 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000003.23.9, , 00000000010.43.5, , 00000000009.32.9, , , , , ] 2020-11-09T12:45:55.447 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:55.440+0000 7faabe3f10c0 0 bucket checkpoint complete 2020-11-09T12:45:55.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:55.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:55.476 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:45:55.518 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:45:55.519 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:45:55.756 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:55.752+0000 7f61480e20c0 1 bucket sync caught up with source: 2020-11-09T12:45:55.757 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000003.19.9, , 00000000003.25.9, , 00000000014.39.5, , , , , ] 2020-11-09T12:45:55.757 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000003.19.9, , 00000000003.25.9, , 00000000014.39.5, , , , , ] 2020-11-09T12:45:55.757 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:45:55.752+0000 7f61480e20c0 0 bucket checkpoint complete 2020-11-09T12:45:55.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:55.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:55.784 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:45:55.832 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:45:55.832 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:45:56.057 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:56.051+0000 7fb71ba060c0 1 bucket sync caught up with source: 2020-11-09T12:45:56.058 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000003.19.9, , 00000000003.25.9, , 00000000014.39.5, , , , , ] 2020-11-09T12:45:56.058 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000003.19.9, , 00000000003.25.9, , 00000000014.39.5, , , , , ] 2020-11-09T12:45:56.058 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:56.051+0000 7fb71ba060c0 0 bucket checkpoint complete 2020-11-09T12:45:56.087 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-17 zones={test-zone2, test-zone1} 2020-11-09T12:45:56.638 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:45:56.646 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:45:56.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:45:56.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:45:56.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:45:56.647 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:45:56.653 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:45:56.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:45:56.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:45:56.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:45:56.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:56.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:56.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:56.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:56.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:56.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:57.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:57.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:57.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:57.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:57.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:57.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:57.633 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-17 zones={test-zone2, test-zone1} 2020-11-09T12:45:57.634 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-17 zones={test-zone3, test-zone1} 2020-11-09T12:45:57.687 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:45:57.693 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:45:57.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:45:57.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:45:57.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:45:57.694 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:45:57.698 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:45:57.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:45:57.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:45:57.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:45:57.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:57.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:57.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:57.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:57.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:58.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:58.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:58.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:58.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:58.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:58.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:58.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:58.657 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-17 zones={test-zone3, test-zone1} 2020-11-09T12:45:58.657 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-17 zones={test-zone3, test-zone2} 2020-11-09T12:45:58.675 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:45:58.682 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:45:58.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:45:58.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:45:58.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:45:58.683 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:45:58.690 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:45:58.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:45:58.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:45:58.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:45:58.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:58.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:58.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:45:58.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:58.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:59.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:45:59.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:59.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:59.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:45:59.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:59.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:59.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:45:59.698 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-17 zones={test-zone3, test-zone2} 2020-11-09T12:45:59.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:59.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:45:59.699 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:45:59.743 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:45:59.744 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:45:59.973 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:59.967+0000 7ff5737190c0 1 bucket sync caught up with source: 2020-11-09T12:45:59.973 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000010.24.5, , 00000000003.8.9, , 00000000007.21.9, , , , , ] 2020-11-09T12:45:59.974 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000010.24.5, , 00000000003.8.9, , 00000000007.21.9, , , , , ] 2020-11-09T12:45:59.974 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:45:59.967+0000 7ff5737190c0 0 bucket checkpoint complete 2020-11-09T12:46:00.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:00.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:00.010 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:00.063 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:00.260 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:00.260+0000 7f4361ff50c0 1 bucket sync caught up with source: 2020-11-09T12:46:00.283 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000010.24.5, , 00000000003.8.9, , 00000000007.21.9, , , , , ] 2020-11-09T12:46:00.284 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000010.24.5, , 00000000003.8.9, , 00000000007.21.9, , , , , ] 2020-11-09T12:46:00.284 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:00.260+0000 7f4361ff50c0 0 bucket checkpoint complete 2020-11-09T12:46:00.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:00.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:00.286 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:00.334 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:00.334 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:00.585 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:00.579+0000 7f9138a200c0 1 bucket sync caught up with source: 2020-11-09T12:46:00.585 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000010.24.5, , 00000000003.8.9, , 00000000007.21.9, , , , , ] 2020-11-09T12:46:00.586 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000010.24.5, , 00000000003.8.9, , 00000000007.21.9, , , , , ] 2020-11-09T12:46:00.586 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:00.579+0000 7f9138a200c0 0 bucket checkpoint complete 2020-11-09T12:46:00.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:00.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:00.704 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:00.969 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:00.967+0000 7fe0117c30c0 1 bucket sync caught up with source: 2020-11-09T12:46:00.970 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000003.15.9, , 00000000010.25.5, , 00000000007.25.9, , , , , ] 2020-11-09T12:46:00.970 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000003.15.9, , 00000000010.25.5, , 00000000007.25.9, , , , , ] 2020-11-09T12:46:00.970 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:00.967+0000 7fe0117c30c0 0 bucket checkpoint complete 2020-11-09T12:46:00.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:00.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:00.999 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:01.255 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:01.251+0000 7f6d6d8f20c0 1 bucket sync caught up with source: 2020-11-09T12:46:01.255 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000003.15.9, , 00000000010.25.5, , 00000000007.25.9, , , , , ] 2020-11-09T12:46:01.255 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000003.15.9, , 00000000010.25.5, , 00000000007.25.9, , , , , ] 2020-11-09T12:46:01.256 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:01.251+0000 7f6d6d8f20c0 0 bucket checkpoint complete 2020-11-09T12:46:01.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:01.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:01.293 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:01.339 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:01.339 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:01.577 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:01.575+0000 7f15bff9b0c0 1 bucket sync caught up with source: 2020-11-09T12:46:01.578 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000003.16.9, , 00000000003.12.9, , 00000000015.42.5, , , , , ] 2020-11-09T12:46:01.578 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000003.16.9, , 00000000003.12.9, , 00000000015.42.5, , , , , ] 2020-11-09T12:46:01.578 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:01.575+0000 7f15bff9b0c0 0 bucket checkpoint complete 2020-11-09T12:46:01.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:01.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:01.606 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:01.654 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:01.654 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:01.894 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:01.887+0000 7f1947ba30c0 1 bucket sync caught up with source: 2020-11-09T12:46:01.894 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000003.16.9, , 00000000003.12.9, , 00000000015.42.5, , , , , ] 2020-11-09T12:46:01.895 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000003.16.9, , 00000000003.12.9, , 00000000015.42.5, , , , , ] 2020-11-09T12:46:01.895 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:01.887+0000 7f1947ba30c0 0 bucket checkpoint complete 2020-11-09T12:46:01.926 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-18 zones={test-zone2, test-zone1} 2020-11-09T12:46:03.370 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:46:03.378 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:03.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:03.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:03.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:03.379 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:46:03.386 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:03.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:03.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:03.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:03.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:03.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:03.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:03.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:03.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:03.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:03.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:03.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:03.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:04.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:04.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:04.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:04.362 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-18 zones={test-zone2, test-zone1} 2020-11-09T12:46:04.362 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-18 zones={test-zone3, test-zone1} 2020-11-09T12:46:04.818 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:46:04.822 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:04.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:04.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:04.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:04.824 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:46:04.829 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:04.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:04.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:04.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:04.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:04.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:04.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:04.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:05.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:05.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:05.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:05.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:05.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:05.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:05.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:05.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:05.821 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-18 zones={test-zone3, test-zone1} 2020-11-09T12:46:05.821 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-18 zones={test-zone3, test-zone2} 2020-11-09T12:46:06.554 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:46:06.565 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:06.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:06.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:06.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:06.567 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:46:06.573 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:06.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:06.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:06.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:06.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:06.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:06.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:06.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:06.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:06.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:07.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:07.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:07.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:07.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:07.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:07.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:07.582 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-18 zones={test-zone3, test-zone2} 2020-11-09T12:46:09.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:09.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:09.677 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:09.728 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:09.729 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:09.944 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:09.938+0000 7ff00b1060c0 1 bucket sync caught up with source: 2020-11-09T12:46:09.944 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000011.33.5, , 00000000004.14.5, , 00000000009.27.5, , , , , ] 2020-11-09T12:46:09.944 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000011.33.5, , 00000000004.14.5, , 00000000009.27.5, , , , , ] 2020-11-09T12:46:09.945 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:09.938+0000 7ff00b1060c0 0 bucket checkpoint complete 2020-11-09T12:46:09.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:09.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:09.971 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:10.018 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:10.019 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:10.229 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:10.227+0000 7f55ed2c40c0 1 bucket sync caught up with source: 2020-11-09T12:46:10.229 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000011.33.5, , 00000000004.14.5, , 00000000009.27.5, , , , , ] 2020-11-09T12:46:10.229 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000011.33.5, , 00000000004.14.5, , 00000000009.27.5, , , , , ] 2020-11-09T12:46:10.230 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:10.227+0000 7f55ed2c40c0 0 bucket checkpoint complete 2020-11-09T12:46:10.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:10.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:10.256 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:10.304 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:10.304 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:10.537 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:10.530+0000 7f95f2e960c0 1 bucket sync caught up with source: 2020-11-09T12:46:10.537 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000011.33.5, , 00000000004.14.5, , 00000000009.27.5, , , , , ] 2020-11-09T12:46:10.537 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000011.33.5, , 00000000004.14.5, , 00000000009.27.5, , , , , ] 2020-11-09T12:46:10.538 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:10.530+0000 7f95f2e960c0 0 bucket checkpoint complete 2020-11-09T12:46:10.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:10.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:10.578 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:10.847 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:10.843+0000 7fa4c616a0c0 1 bucket sync caught up with source: 2020-11-09T12:46:10.848 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000004.16.5, , 00000000011.35.5, , 00000000008.32.5, , , , , ] 2020-11-09T12:46:10.848 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000004.16.5, , 00000000011.35.5, , 00000000008.32.5, , , , , ] 2020-11-09T12:46:10.848 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:10.843+0000 7fa4c616a0c0 0 bucket checkpoint complete 2020-11-09T12:46:10.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:10.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:10.877 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:10.922 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:10.923 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:11.139 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:11.135+0000 7f43ad8f20c0 1 bucket sync caught up with source: 2020-11-09T12:46:11.140 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000004.16.5, , 00000000011.35.5, , 00000000008.32.5, , , , , ] 2020-11-09T12:46:11.140 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000004.16.5, , 00000000011.35.5, , 00000000008.32.5, , , , , ] 2020-11-09T12:46:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:11.135+0000 7f43ad8f20c0 0 bucket checkpoint complete 2020-11-09T12:46:11.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:11.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:11.172 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:11.258 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:11.259 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:11.489 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:11.487+0000 7facbcb8f0c0 1 bucket sync caught up with source: 2020-11-09T12:46:11.489 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000004.17.5, , 00000000004.20.5, , 00000000015.44.5, , , , , ] 2020-11-09T12:46:11.489 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000004.17.5, , 00000000004.20.5, , 00000000015.44.5, , , , , ] 2020-11-09T12:46:11.489 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:11.487+0000 7facbcb8f0c0 0 bucket checkpoint complete 2020-11-09T12:46:11.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:11.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:11.515 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:11.561 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:11.562 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:11.805 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:11.798+0000 7fc0b40840c0 1 bucket sync caught up with source: 2020-11-09T12:46:11.805 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000004.17.5, , 00000000004.20.5, , 00000000015.44.5, , , , , ] 2020-11-09T12:46:11.805 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000004.17.5, , 00000000004.20.5, , 00000000015.44.5, , , , , ] 2020-11-09T12:46:11.805 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:11.798+0000 7fc0b40840c0 0 bucket checkpoint complete 2020-11-09T12:46:11.832 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-16 zones={test-zone2, test-zone1} 2020-11-09T12:46:13.188 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:46:13.196 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:13.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:13.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:13.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:13.197 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:46:13.205 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:13.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:13.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:13.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:13.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:13.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:13.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:13.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:13.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:13.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:13.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:13.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:13.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:13.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:14.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:14.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:14.237 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-16 zones={test-zone2, test-zone1} 2020-11-09T12:46:14.238 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-16 zones={test-zone3, test-zone1} 2020-11-09T12:46:14.446 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:46:14.452 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:14.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:14.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:14.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:14.453 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:46:14.458 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:14.458 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:14.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:14.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:14.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:14.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:14.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:14.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:14.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:14.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:14.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:14.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:15.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:15.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:15.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:15.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:15.433 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-16 zones={test-zone3, test-zone1} 2020-11-09T12:46:15.434 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-16 zones={test-zone3, test-zone2} 2020-11-09T12:46:16.414 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:46:16.422 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:16.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:16.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:16.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:16.424 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:46:16.433 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:16.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:16.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:16.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:16.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:16.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:16.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:16.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:16.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:16.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:16.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:16.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:17.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:17.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:17.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:17.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:17.425 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-16 zones={test-zone3, test-zone2} 2020-11-09T12:46:17.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:17.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:17.426 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:17.470 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:17.470 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:17.691 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:17.683+0000 7fbd97bd50c0 1 bucket sync caught up with source: 2020-11-09T12:46:17.692 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000011.42.5, , 00000000004.44.5, , 00000000008.25.5, , , , , ] 2020-11-09T12:46:17.692 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000011.42.5, , 00000000004.44.5, , 00000000008.25.5, , , , , ] 2020-11-09T12:46:17.692 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:17.683+0000 7fbd97bd50c0 0 bucket checkpoint complete 2020-11-09T12:46:17.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:17.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:17.723 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:17.769 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:17.769 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:18.010 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:18.007+0000 7fed2e20a0c0 1 bucket sync caught up with source: 2020-11-09T12:46:18.011 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000011.42.5, , 00000000004.44.5, , 00000000008.25.5, , , , , ] 2020-11-09T12:46:18.011 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000011.42.5, , 00000000004.44.5, , 00000000008.25.5, , , , , ] 2020-11-09T12:46:18.011 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:18.007+0000 7fed2e20a0c0 0 bucket checkpoint complete 2020-11-09T12:46:18.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:18.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:18.040 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:18.088 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:18.089 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:18.337 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:18.331+0000 7f1a5053c0c0 1 bucket sync caught up with source: 2020-11-09T12:46:18.337 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000011.42.5, , 00000000004.44.5, , 00000000008.25.5, , , , , ] 2020-11-09T12:46:18.337 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000011.42.5, , 00000000004.44.5, , 00000000008.25.5, , , , , ] 2020-11-09T12:46:18.337 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:18.331+0000 7f1a5053c0c0 0 bucket checkpoint complete 2020-11-09T12:46:18.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:18.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:18.457 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:18.504 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:18.505 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:18.748 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:18.747+0000 7fea3ae550c0 1 bucket sync caught up with source: 2020-11-09T12:46:18.749 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000004.52.5, , 00000000011.50.5, , 00000000010.35.5, , , , , ] 2020-11-09T12:46:18.749 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000004.52.5, , 00000000011.50.5, , 00000000010.35.5, , , , , ] 2020-11-09T12:46:18.749 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:18.747+0000 7fea3ae550c0 0 bucket checkpoint complete 2020-11-09T12:46:18.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:18.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:18.780 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:18.826 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:18.827 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:19.050 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:19.047+0000 7fb73c5de0c0 1 bucket sync caught up with source: 2020-11-09T12:46:19.051 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000004.52.5, , 00000000011.50.5, , 00000000010.35.5, , , , , ] 2020-11-09T12:46:19.051 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000004.52.5, , 00000000011.50.5, , 00000000010.35.5, , , , , ] 2020-11-09T12:46:19.051 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:19.047+0000 7fb73c5de0c0 0 bucket checkpoint complete 2020-11-09T12:46:19.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:19.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:19.078 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:19.120 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:19.121 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:19.345 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:19.343+0000 7f702c0470c0 1 bucket sync caught up with source: 2020-11-09T12:46:19.346 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000004.30.5, , 00000000004.28.5, , 00000000015.42.5, , , , , ] 2020-11-09T12:46:19.346 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000004.30.5, , 00000000004.28.5, , 00000000015.42.5, , , , , ] 2020-11-09T12:46:19.346 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:19.343+0000 7f702c0470c0 0 bucket checkpoint complete 2020-11-09T12:46:19.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:19.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:19.371 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:19.420 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:19.421 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:19.658 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:19.651+0000 7f03120710c0 1 bucket sync caught up with source: 2020-11-09T12:46:19.659 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000004.30.5, , 00000000004.28.5, , 00000000015.42.5, , , , , ] 2020-11-09T12:46:19.659 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000004.30.5, , 00000000004.28.5, , 00000000015.42.5, , , , , ] 2020-11-09T12:46:19.659 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:19.651+0000 7f03120710c0 0 bucket checkpoint complete 2020-11-09T12:46:19.689 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-17 zones={test-zone2, test-zone1} 2020-11-09T12:46:20.624 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:46:20.631 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:20.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:20.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:20.633 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:20.633 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:46:20.639 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:20.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:20.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:20.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:20.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:20.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:20.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:20.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:20.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:20.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:21.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:21.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:21.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:21.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:21.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:21.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:21.641 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-17 zones={test-zone2, test-zone1} 2020-11-09T12:46:21.642 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-17 zones={test-zone3, test-zone1} 2020-11-09T12:46:22.009 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:46:22.015 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:22.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:22.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:22.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:22.018 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:46:22.023 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:22.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:22.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:22.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:22.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:22.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:22.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:22.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:22.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:22.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:22.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:22.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:22.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:22.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:22.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:22.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:23.001 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-17 zones={test-zone3, test-zone1} 2020-11-09T12:46:23.002 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-17 zones={test-zone3, test-zone2} 2020-11-09T12:46:23.893 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:46:23.902 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:23.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:23.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:23.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:23.904 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:46:23.912 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:23.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:23.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:23.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:23.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:23.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:24.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:24.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:24.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:24.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:24.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:24.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:24.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:24.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:24.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:24.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:24.909 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-17 zones={test-zone3, test-zone2} 2020-11-09T12:46:24.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:24.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:24.911 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:24.956 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:24.956 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:25.192 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:25.187+0000 7f78170740c0 1 bucket sync caught up with source: 2020-11-09T12:46:25.192 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000011.30.5, , 00000000004.11.5, , 00000000008.24.5, , , , , ] 2020-11-09T12:46:25.192 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000011.30.5, , 00000000004.11.5, , 00000000008.24.5, , , , , ] 2020-11-09T12:46:25.192 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:25.187+0000 7f78170740c0 0 bucket checkpoint complete 2020-11-09T12:46:25.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:25.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:25.224 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:25.271 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:25.271 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:25.498 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:25.495+0000 7f8fac8590c0 1 bucket sync caught up with source: 2020-11-09T12:46:25.499 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000011.30.5, , 00000000004.11.5, , 00000000008.24.5, , , , , ] 2020-11-09T12:46:25.499 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000011.30.5, , 00000000004.11.5, , 00000000008.24.5, , , , , ] 2020-11-09T12:46:25.499 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:25.495+0000 7f8fac8590c0 0 bucket checkpoint complete 2020-11-09T12:46:25.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:25.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:25.526 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:25.571 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:25.572 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:25.799 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:25.791+0000 7f26aedab0c0 1 bucket sync caught up with source: 2020-11-09T12:46:25.799 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000011.30.5, , 00000000004.11.5, , 00000000008.24.5, , , , , ] 2020-11-09T12:46:25.800 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000011.30.5, , 00000000004.11.5, , 00000000008.24.5, , , , , ] 2020-11-09T12:46:25.800 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:25.791+0000 7f26aedab0c0 0 bucket checkpoint complete 2020-11-09T12:46:25.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:25.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:25.939 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:25.986 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:25.986 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:26.207 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:26.203+0000 7fc7c81c70c0 1 bucket sync caught up with source: 2020-11-09T12:46:26.207 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000004.18.5, , 00000000011.32.5, , 00000000008.28.5, , , , , ] 2020-11-09T12:46:26.207 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000004.18.5, , 00000000011.32.5, , 00000000008.28.5, , , , , ] 2020-11-09T12:46:26.208 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:26.203+0000 7fc7c81c70c0 0 bucket checkpoint complete 2020-11-09T12:46:26.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:26.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:26.237 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:26.283 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:26.283 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:26.499 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:26.495+0000 7f47fe4c10c0 1 bucket sync caught up with source: 2020-11-09T12:46:26.499 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000004.18.5, , 00000000011.32.5, , 00000000008.28.5, , , , , ] 2020-11-09T12:46:26.499 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000004.18.5, , 00000000011.32.5, , 00000000008.28.5, , , , , ] 2020-11-09T12:46:26.500 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:26.495+0000 7f47fe4c10c0 0 bucket checkpoint complete 2020-11-09T12:46:26.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:26.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:26.529 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:26.847 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:26.843+0000 7f028544e0c0 1 bucket sync caught up with source: 2020-11-09T12:46:26.847 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, 00000000004.19.5, , 00000000004.15.5, , 00000000016.45.5, , , , , ] 2020-11-09T12:46:26.847 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, 00000000004.19.5, , 00000000004.15.5, , 00000000016.45.5, , , , , ] 2020-11-09T12:46:26.848 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:26.843+0000 7f028544e0c0 0 bucket checkpoint complete 2020-11-09T12:46:26.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:26.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:26.879 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:26.929 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:26.930 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:27.162 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:27.155+0000 7f50e17c90c0 1 bucket sync caught up with source: 2020-11-09T12:46:27.162 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, 00000000004.19.5, , 00000000004.15.5, , 00000000016.45.5, , , , , ] 2020-11-09T12:46:27.162 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, 00000000004.19.5, , 00000000004.15.5, , 00000000016.45.5, , , , , ] 2020-11-09T12:46:27.163 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:27.155+0000 7f50e17c90c0 0 bucket checkpoint complete 2020-11-09T12:46:27.195 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-18 zones={test-zone2, test-zone1} 2020-11-09T12:46:27.908 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:46:27.916 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:27.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:27.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:27.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:27.917 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:46:27.925 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:27.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:27.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:27.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:27.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:27.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:28.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:28.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:28.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:28.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:28.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:28.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:28.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:28.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:28.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:28.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:28.910 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-18 zones={test-zone2, test-zone1} 2020-11-09T12:46:28.911 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-18 zones={test-zone3, test-zone1} 2020-11-09T12:46:28.970 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:46:28.975 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:28.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:28.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:28.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:28.976 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:46:28.981 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:28.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:28.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:28.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:28.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:28.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:29.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:29.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:29.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:29.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:29.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:29.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:29.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:29.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:29.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:29.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:29.941 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-18 zones={test-zone3, test-zone1} 2020-11-09T12:46:29.942 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-18 zones={test-zone3, test-zone2} 2020-11-09T12:46:30.017 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:46:30.025 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:30.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:30.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:30.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:30.026 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:46:30.032 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:46:30.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-09T12:46:30.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-09T12:46:30.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-09T12:46:30.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:30.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:30.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:46:30.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:30.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:30.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-09T12:46:30.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:30.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:30.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-09T12:46:30.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:30.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:30.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-09T12:46:31.006 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-18 zones={test-zone3, test-zone2} 2020-11-09T12:46:31.007 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2020-11-09T12:46:31.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:46:31.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:46:31.225 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:46:31.261 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:31.261 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:31.467 INFO:teuthology.orchestra.run.smithi164.stderr:No --period given, using current period=d8c95da7-75a8-4dcf-94ba-f12d5c80a46f 2020-11-09T12:46:31.502 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:46:31.502 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:46:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925991.221284_94.1", 2020-11-09T12:46:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:46:31.221284Z" 2020-11-09T12:46:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:46:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:46:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:46:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:42:16.653632Z" 2020-11-09T12:46:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:46:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:46:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:46:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:46:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:46:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:46:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:46:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:46:31.505 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:46:31.505 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:46:31.506 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T12:46:31.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:46:31.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:46:31.507 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:46:31.550 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:31.550 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:31.818 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:46:31.819 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:46:31.819 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:46:31.819 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:46:31.819 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:46:31.820 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:46:31.820 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:46:31.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:46:31.820 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:46:31.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:46:31.821 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:46:31.821 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:46:31.821 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:46:31.821 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925991.221284_94.1", 2020-11-09T12:46:31.821 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:46:31.821 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:46:31.822 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:46:31.822 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:46:31.221284Z", 2020-11-09T12:46:31.822 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:46:31.822 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:46:31.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:46:31.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:46:31.823 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:46:31.823 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:46:31.823 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:46:31.823 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:46:31.823 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:46:31.823 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:46:31.823 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:46:31.824 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:46:31.824 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:46:31.824 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:46:31.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:46:31.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:46:31.824 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:46:31.824 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:46:31.825 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:46:31.825 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:46:31.825 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:46:31.825 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:46:31.825 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:46:31.825 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:46:31.825 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:46:31.826 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:46:31.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:46:31.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:46:31.826 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:46:31.826 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:46:31.826 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:46:31.826 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:46:31.827 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:46:31.827 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:46:31.827 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:46:31.827 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:46:31.827 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:46:31.827 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:46:31.827 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:46:31.827 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:46:31.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:46:31.828 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:46:31.828 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:46:31.828 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:46:31.828 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:46:31.828 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:46:31.829 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T12:46:31.830 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T12:46:31.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:46:31.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:46:31.831 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:46:31.870 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:31.870 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:32.097 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:46:32.098 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:46:32.098 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:46:32.098 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:46:32.098 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:46:32.098 INFO:teuthology.orchestra.run.smithi164.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:46:32.098 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:46:32.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:46:32.099 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:46:32.099 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:46:32.099 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:46:32.099 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:46:32.099 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:46:32.099 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925991.221284_94.1", 2020-11-09T12:46:32.100 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:46:32.100 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 1, 2020-11-09T12:46:32.100 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:46:32.100 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:46:31.221284Z", 2020-11-09T12:46:32.100 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:46:32.100 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:46:32.100 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:46:32.101 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:46:32.101 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:46:32.101 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:46:32.101 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:46:32.101 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:46:32.101 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:46:32.101 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:46:32.101 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:46:32.102 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:46:32.102 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:46:32.102 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:46:32.102 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:46:32.102 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:46:32.103 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:46:32.103 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:46:32.103 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:46:32.103 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:46:32.103 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:46:32.103 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:46:32.104 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:46:32.104 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:46:32.104 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:46:32.104 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:46:32.104 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:46:32.104 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:46:32.104 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:46:32.104 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:46:32.105 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:46:32.105 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:46:32.105 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:46:32.105 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:46:32.105 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:46:32.105 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:46:32.105 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:46:32.106 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:46:32.106 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:46:32.106 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:46:32.106 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:46:32.106 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:46:32.106 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 1, 2020-11-09T12:46:32.106 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 1 2020-11-09T12:46:32.107 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:46:32.107 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:46:32.108 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T12:46:32.109 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T12:46:32.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:46:32.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:46:32.110 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:46:32.151 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:32.151 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:32.413 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:46:32.413 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:46:32.414 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:46:32.414 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:46:32.414 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:46:32.414 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:46:32.414 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:46:32.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:46:32.415 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:46:32.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:46:32.415 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:46:32.415 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:46:32.415 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:46:32.415 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925991.221284_94.1", 2020-11-09T12:46:32.416 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:46:32.416 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:46:32.416 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:46:32.416 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:46:31.221284Z", 2020-11-09T12:46:32.416 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:46:32.416 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:46:32.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:46:32.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:46:32.417 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:46:32.417 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:46:32.417 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:46:32.417 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:46:32.417 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:46:32.418 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:46:32.418 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:46:32.418 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:46:32.418 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:46:32.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:46:32.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:46:32.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:46:32.419 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:46:32.419 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:46:32.419 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:46:32.419 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:46:32.419 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:46:32.420 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:46:32.420 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:46:32.420 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:46:32.420 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:46:32.420 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:46:32.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:46:32.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:46:32.421 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:46:32.421 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:46:32.421 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:46:32.421 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:46:32.421 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:46:32.421 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:46:32.422 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:46:32.422 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:46:32.422 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:46:32.422 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:46:32.422 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:46:32.423 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:46:32.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:46:32.423 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:46:32.423 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:46:32.423 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:46:32.423 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:46:32.424 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:46:32.425 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T12:46:32.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:32.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:32.823 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:32.871 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:32.872 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:33.094 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:33.087+0000 7fdd1a7500c0 1 bucket sync caught up with source: 2020-11-09T12:46:33.095 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-09T12:46:33.095 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000002.10.10, , , , , ] 2020-11-09T12:46:33.095 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:33.087+0000 7fdd1a7500c0 0 bucket checkpoint complete 2020-11-09T12:46:33.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:33.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:33.123 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:33.169 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:33.169 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:33.390 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:33.387+0000 7ff6617bf0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:46:33.390 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-11-09T12:46:33.391 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.14.9, , , , , ] 2020-11-09T12:46:38.394 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:38.391+0000 7ff6617bf0c0 1 bucket sync caught up with source: 2020-11-09T12:46:38.395 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000005.16.9, , , , , ] 2020-11-09T12:46:38.395 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.14.9, , , , , ] 2020-11-09T12:46:38.395 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:38.391+0000 7ff6617bf0c0 0 bucket checkpoint complete 2020-11-09T12:46:38.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:38.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:38.425 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:38.473 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:38.474 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:38.698 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:38.691+0000 7fc63099f0c0 1 bucket sync caught up with source: 2020-11-09T12:46:38.698 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000005.16.9, , , , , ] 2020-11-09T12:46:38.699 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-11-09T12:46:38.699 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:38.691+0000 7fc63099f0c0 0 bucket checkpoint complete 2020-11-09T12:46:38.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:38.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:38.739 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:38.787 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:38.787 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:39.026 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:39.023+0000 7f6210b6f0c0 1 bucket sync caught up with source: 2020-11-09T12:46:39.026 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000004.18.9, , , , , ] 2020-11-09T12:46:39.026 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.18.9, , , , , ] 2020-11-09T12:46:39.027 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:39.023+0000 7f6210b6f0c0 0 bucket checkpoint complete 2020-11-09T12:46:39.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:39.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:39.056 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:39.105 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:39.105 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:39.341 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:39.339+0000 7fbe5f3600c0 1 bucket sync caught up with source: 2020-11-09T12:46:39.341 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000004.18.9, , , , , ] 2020-11-09T12:46:39.341 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.18.9, , , , , ] 2020-11-09T12:46:39.341 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:39.339+0000 7fbe5f3600c0 0 bucket checkpoint complete 2020-11-09T12:46:39.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:39.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:39.370 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:39.417 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:39.417 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:39.648 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:39.647+0000 7fcf82cdb0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:46:39.648 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000003.17.9, , , , , ] 2020-11-09T12:46:39.648 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2020-11-09T12:46:44.652 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:44.647+0000 7fcf82cdb0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:46:44.652 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000003.17.9, , , , , ] 2020-11-09T12:46:44.652 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2020-11-09T12:46:49.656 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:49.651+0000 7fcf82cdb0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:46:49.656 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000003.17.9, , , , , ] 2020-11-09T12:46:49.656 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2020-11-09T12:46:54.659 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:54.655+0000 7fcf82cdb0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:46:54.659 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000003.17.9, , , , , ] 2020-11-09T12:46:54.660 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2020-11-09T12:46:59.664 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:59.659+0000 7fcf82cdb0c0 1 bucket sync caught up with source: 2020-11-09T12:46:59.664 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000004.19.9, , , , , ] 2020-11-09T12:46:59.664 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2020-11-09T12:46:59.664 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:46:59.659+0000 7fcf82cdb0c0 0 bucket checkpoint complete 2020-11-09T12:46:59.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:59.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:46:59.697 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:46:59.744 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:46:59.745 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:46:59.961 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:59.955+0000 7f21a171e0c0 1 bucket sync caught up with source: 2020-11-09T12:46:59.962 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000004.19.9, , , , , ] 2020-11-09T12:46:59.962 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2020-11-09T12:46:59.962 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:46:59.955+0000 7f21a171e0c0 0 bucket checkpoint complete 2020-11-09T12:46:59.992 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-19 zones={test-zone2, test-zone1} 2020-11-09T12:47:00.902 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:47:00.910 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:47:00.911 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:47:00.958 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:47:01.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:47:01.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:47:01.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:47:01.153 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-19 zones={test-zone2, test-zone1} 2020-11-09T12:47:01.153 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-19 zones={test-zone3, test-zone1} 2020-11-09T12:47:01.719 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:47:01.766 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:47:01.767 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:47:01.774 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:47:01.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:47:01.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:47:01.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:47:01.961 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-19 zones={test-zone3, test-zone1} 2020-11-09T12:47:01.961 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-19 zones={test-zone3, test-zone2} 2020-11-09T12:47:01.971 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:47:02.018 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:47:02.019 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:47:02.026 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:47:02.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:47:02.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:47:02.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:47:02.225 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-19 zones={test-zone3, test-zone2} 2020-11-09T12:47:03.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:47:03.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:47:03.736 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:47:03.783 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:47:03.783 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:47:04.033 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:47:04.027+0000 7fd69dcd20c0 1 waiting for incremental sync to catch up: 2020-11-09T12:47:04.034 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000005.16.9, , , 00000000004.15.11, , ] 2020-11-09T12:47:04.034 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.16.9, , , 00000000031.87.11, , ] 2020-11-09T12:47:09.038 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:47:09.031+0000 7fd69dcd20c0 1 waiting for incremental sync to catch up: 2020-11-09T12:47:09.038 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000005.16.9, , , 00000000004.15.11, , ] 2020-11-09T12:47:09.038 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.16.9, , , 00000000031.87.11, , ] 2020-11-09T12:47:14.041 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:47:14.035+0000 7fd69dcd20c0 1 waiting for incremental sync to catch up: 2020-11-09T12:47:14.041 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000005.16.9, , , 00000000004.15.11, , ] 2020-11-09T12:47:14.042 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.16.9, , , 00000000031.87.11, , ] 2020-11-09T12:47:19.044 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:47:19.039+0000 7fd69dcd20c0 1 waiting for incremental sync to catch up: 2020-11-09T12:47:19.044 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000005.16.9, , , 00000000004.15.11, , ] 2020-11-09T12:47:19.045 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.16.9, , , 00000000031.87.11, , ] 2020-11-09T12:47:24.048 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:47:24.043+0000 7fd69dcd20c0 1 bucket sync caught up with source: 2020-11-09T12:47:24.048 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000005.16.9, , , 00000000035.100.5, , ] 2020-11-09T12:47:24.048 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.16.9, , , 00000000031.87.11, , ] 2020-11-09T12:47:24.049 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:47:24.043+0000 7fd69dcd20c0 0 bucket checkpoint complete 2020-11-09T12:47:24.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:47:24.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:47:24.079 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:47:24.129 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:47:24.130 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:47:24.346 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:47:24.342+0000 7fcb73a400c0 1 bucket sync caught up with source: 2020-11-09T12:47:24.347 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000005.16.9, , , 00000000045.125.9, , ] 2020-11-09T12:47:24.347 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000005.16.9, , , 00000000045.125.9, , ] 2020-11-09T12:47:24.347 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:47:24.342+0000 7fcb73a400c0 0 bucket checkpoint complete 2020-11-09T12:47:24.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:47:24.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:47:24.375 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:47:24.422 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:47:24.422 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:47:24.653 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:47:24.651+0000 7fa4c659f0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:47:24.654 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000005.16.9, , , 00000000031.87.11, , ] 2020-11-09T12:47:24.654 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.16.9, , , 00000000045.125.9, , ] 2020-11-09T12:47:29.657 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:47:29.655+0000 7fa4c659f0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:47:29.658 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000005.16.9, , , 00000000031.87.11, , ] 2020-11-09T12:47:29.658 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.16.9, , , 00000000045.125.9, , ] 2020-11-09T12:47:34.661 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:47:34.659+0000 7fa4c659f0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:47:34.662 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000005.16.9, , , 00000000031.87.11, , ] 2020-11-09T12:47:34.662 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.16.9, , , 00000000045.125.9, , ] 2020-11-09T12:47:39.665 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:47:39.663+0000 7fa4c659f0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:47:39.666 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000005.16.9, , , 00000000031.87.11, , ] 2020-11-09T12:47:39.666 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.16.9, , , 00000000045.125.9, , ] 2020-11-09T12:47:44.669 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:47:44.667+0000 7fa4c659f0c0 1 bucket sync caught up with source: 2020-11-09T12:47:44.669 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000005.16.9, , , 00000000045.125.9, , ] 2020-11-09T12:47:44.670 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000005.16.9, , , 00000000045.125.9, , ] 2020-11-09T12:47:44.670 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:47:44.667+0000 7fa4c659f0c0 0 bucket checkpoint complete 2020-11-09T12:47:44.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:47:44.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:47:44.812 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:47:44.859 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:47:44.860 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:47:45.102 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:47:45.098+0000 7f163a9e30c0 1 waiting for incremental sync to catch up: 2020-11-09T12:47:45.103 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000046.127.8, , ] 2020-11-09T12:47:45.103 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000048.131.9, , ] 2020-11-09T12:47:50.107 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:47:50.102+0000 7f163a9e30c0 1 waiting for incremental sync to catch up: 2020-11-09T12:47:50.107 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000046.127.8, , ] 2020-11-09T12:47:50.108 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000048.131.9, , ] 2020-11-09T12:47:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:47:55.106+0000 7f163a9e30c0 1 waiting for incremental sync to catch up: 2020-11-09T12:47:55.111 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000046.127.8, , ] 2020-11-09T12:47:55.111 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000048.131.9, , ] 2020-11-09T12:48:00.114 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:48:00.110+0000 7f163a9e30c0 1 waiting for incremental sync to catch up: 2020-11-09T12:48:00.115 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000046.127.8, , ] 2020-11-09T12:48:00.115 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000048.131.9, , ] 2020-11-09T12:48:05.119 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:48:05.114+0000 7f163a9e30c0 1 waiting for incremental sync to catch up: 2020-11-09T12:48:05.120 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000046.127.8, , ] 2020-11-09T12:48:05.120 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000048.131.9, , ] 2020-11-09T12:48:10.123 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:48:10.118+0000 7f163a9e30c0 1 waiting for incremental sync to catch up: 2020-11-09T12:48:10.124 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000046.127.8, , ] 2020-11-09T12:48:10.124 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000048.131.9, , ] 2020-11-09T12:48:15.126 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:48:15.122+0000 7f163a9e30c0 1 waiting for incremental sync to catch up: 2020-11-09T12:48:15.127 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000046.127.8, , ] 2020-11-09T12:48:15.127 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000048.131.9, , ] 2020-11-09T12:48:20.129 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:48:20.126+0000 7f163a9e30c0 1 bucket sync caught up with source: 2020-11-09T12:48:20.130 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000048.131.9, , ] 2020-11-09T12:48:20.130 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000048.131.9, , ] 2020-11-09T12:48:20.130 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:48:20.126+0000 7f163a9e30c0 0 bucket checkpoint complete 2020-11-09T12:48:20.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:48:20.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:48:20.162 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:48:20.211 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:20.212 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:20.433 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:48:20.430+0000 7f30256570c0 1 bucket sync caught up with source: 2020-11-09T12:48:20.433 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000048.131.9, , ] 2020-11-09T12:48:20.434 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000048.131.9, , ] 2020-11-09T12:48:20.434 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:48:20.430+0000 7f30256570c0 0 bucket checkpoint complete 2020-11-09T12:48:20.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:48:20.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:48:20.464 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:48:20.552 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:20.552 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:20.802 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:48:20.798+0000 7f31770740c0 1 bucket sync caught up with source: 2020-11-09T12:48:20.802 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000052.134.9, , ] 2020-11-09T12:48:20.802 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000052.134.9, , ] 2020-11-09T12:48:20.803 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:48:20.798+0000 7f31770740c0 0 bucket checkpoint complete 2020-11-09T12:48:20.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:48:20.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:48:20.832 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:48:20.880 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:20.880 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:21.132 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:48:21.131+0000 7fbd7bb620c0 1 bucket sync caught up with source: 2020-11-09T12:48:21.132 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000052.134.9, , ] 2020-11-09T12:48:21.132 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000052.134.9, , ] 2020-11-09T12:48:21.133 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:48:21.131+0000 7fbd7bb620c0 0 bucket checkpoint complete 2020-11-09T12:48:21.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-19 zones={test-zone2, test-zone1} 2020-11-09T12:48:21.178 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:48:21.189 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:48:21.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.199 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:48:21.207 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:48:21.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:21.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:48:21.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:48:21.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:48:21.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:21.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:21.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:21.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:21.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:21.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:21.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:22.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:22.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:22.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:22.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:22.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:22.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:22.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:22.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:22.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:22.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:22.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:23.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:23.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:23.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:23.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:23.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:23.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:23.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:23.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:23.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:23.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:23.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:24.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:24.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:24.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:24.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:24.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:24.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:24.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:24.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:24.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:24.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:25.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:25.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:25.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:25.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:25.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:25.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:25.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:25.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:25.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:25.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:25.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:26.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:26.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:26.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:26.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:26.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:26.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:26.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:26.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:26.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:26.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:27.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:27.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:27.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:27.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:27.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:27.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:27.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:27.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:27.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:27.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:27.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:28.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:28.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:28.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:28.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:28.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:28.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:28.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:28.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:28.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:28.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:29.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:29.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:29.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:29.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:29.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:29.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:29.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:29.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:29.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:29.845 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-19 zones={test-zone2, test-zone1} 2020-11-09T12:48:29.846 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-19 zones={test-zone3, test-zone1} 2020-11-09T12:48:29.855 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:48:29.861 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:48:29.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.871 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:48:29.879 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:48:29.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:29.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:48:29.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:48:29.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:48:30.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:30.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:30.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:30.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:30.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:30.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:30.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:30.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:30.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:30.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:30.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:31.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:31.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:31.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:31.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:31.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:31.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:31.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:31.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:31.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:31.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:31.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:32.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:32.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:32.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:32.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:32.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:32.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:32.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:32.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:32.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:32.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:33.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:33.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:33.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:33.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:33.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:33.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:33.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:33.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:33.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:33.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:33.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:34.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:34.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:34.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:34.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:34.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:34.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:34.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:34.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:34.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:34.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:35.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:35.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:35.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:35.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:35.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:35.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:35.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:35.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:35.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:35.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:35.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:36.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:36.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:36.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:36.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:36.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:36.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:36.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:36.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:36.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:36.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:36.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:37.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:37.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:37.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:37.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:37.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:37.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:37.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:37.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:37.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:38.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:38.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:38.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:38.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:38.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:38.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:38.566 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-19 zones={test-zone3, test-zone1} 2020-11-09T12:48:38.566 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kuljus-19 zones={test-zone3, test-zone2} 2020-11-09T12:48:38.973 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-09T12:48:38.985 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:48:38.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:38.996 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-09T12:48:39.007 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-09T12:48:39.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-09T12:48:39.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:48:39.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:48:39.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-09T12:48:39.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:39.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:39.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:39.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:39.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:39.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:39.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:39.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:39.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:40.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:40.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:40.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:40.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:40.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:40.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:40.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:40.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:40.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:40.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:40.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:41.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:41.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:41.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:41.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:41.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:41.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:41.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:41.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:41.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:41.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:42.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:42.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:42.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:42.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:42.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:42.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:42.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:42.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:42.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:42.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:43.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:43.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:43.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:43.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:43.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:43.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:43.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:43.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:43.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:43.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:44.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:44.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:44.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:44.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:44.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:44.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:44.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:44.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:44.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:44.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:44.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:45.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:45.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:45.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:45.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:45.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:45.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:45.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:45.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:45.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:45.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:46.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:46.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:46.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:46.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:46.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:46.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:46.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:46.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:46.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:46.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:47.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:47.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:47.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:47.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:47.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:47.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:47.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:47.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:47.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-09T12:48:47.911 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kuljus-19 zones={test-zone3, test-zone2} 2020-11-09T12:48:47.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:47.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:47.914 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:47.954 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:47.954 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:48.183 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:48:48.183 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:48.183 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-09T12:48:48.183 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:46:56.199479Z" 2020-11-09T12:48:48.184 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:48.184 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:48.184 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T12:48:48.184 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:48:48.184 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:48.184 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:48.184 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-09T12:48:48.185 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:46:09.701104Z" 2020-11-09T12:48:48.185 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:48.185 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:48.185 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-09T12:48:48.185 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:48:02.212603Z" 2020-11-09T12:48:48.185 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:48.186 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:48:48.187 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-09T12:48:48.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:48.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:48.189 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:48.230 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:48.230 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:48.486 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:48:48.487 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:48:48.487 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:48:48.487 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:48:48.487 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:48:48.487 INFO:teuthology.orchestra.run.smithi190.stdout: "instance_id": 37223426371803585 2020-11-09T12:48:48.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:48.488 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:48:48.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:48.488 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:48:48.488 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:48.488 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "incremental-sync", 2020-11-09T12:48:48.489 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-09T12:48:48.489 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:48.489 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:48.489 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:48.489 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:46:56.199479Z" 2020-11-09T12:48:48.489 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:48.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:48.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:48.490 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:48:48.490 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:48.490 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "incremental-sync", 2020-11-09T12:48:48.490 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T12:48:48.490 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:48.491 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:48.491 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:48.491 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000" 2020-11-09T12:48:48.491 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:48.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:48.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:48.491 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:48:48.492 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:48.492 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "incremental-sync", 2020-11-09T12:48:48.492 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-09T12:48:48.492 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:48.492 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:48.492 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:48.493 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:46:09.701104Z" 2020-11-09T12:48:48.493 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:48.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:48.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:48.493 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:48:48.493 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:48.493 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "incremental-sync", 2020-11-09T12:48:48.493 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-09T12:48:48.494 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:48.494 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:48.494 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:48.494 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:48:02.212603Z" 2020-11-09T12:48:48.494 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:48.494 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:48.494 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:48:48.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:48.495 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:48:48.495 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 0, 2020-11-09T12:48:48.495 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 0 2020-11-09T12:48:48.495 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:48.495 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:48:48.496 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-09T12:48:48.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:48.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:48.498 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:48.536 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:48.537 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:48.783 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:48:48.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:48.783 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-09T12:48:48.784 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:46:56.199479Z" 2020-11-09T12:48:48.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:48.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:48.784 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T12:48:48.784 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:48:48.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:48.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:48.785 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-09T12:48:48.785 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:46:09.701104Z" 2020-11-09T12:48:48.785 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:48.785 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:48.785 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-09T12:48:48.785 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:48:02.212603Z" 2020-11-09T12:48:48.785 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:48.786 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:48:48.787 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-09T12:48:48.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:48.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:48.788 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:48.872 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:48.873 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:49.105 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:48:49.105 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:48:49.106 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:48:49.106 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:48:49.106 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:48:49.106 INFO:teuthology.orchestra.run.smithi164.stdout: "instance_id": 17658500712352186535 2020-11-09T12:48:49.106 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:49.106 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:48:49.106 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:49.107 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:48:49.107 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:49.107 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:49.107 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-09T12:48:49.107 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:49.107 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:49.108 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:49.108 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:46:56.199479Z" 2020-11-09T12:48:49.108 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:49.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:49.108 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:49.108 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:48:49.108 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:49.109 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:49.109 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T12:48:49.109 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:49.109 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:49.109 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:49.109 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000" 2020-11-09T12:48:49.109 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:49.110 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:49.110 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:49.110 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:48:49.110 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:49.110 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:49.110 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-09T12:48:49.110 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:49.111 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:49.111 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:49.111 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:46:09.701104Z" 2020-11-09T12:48:49.111 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:49.111 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:49.111 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:49.111 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:48:49.112 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:49.112 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:49.112 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-09T12:48:49.112 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:49.112 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:49.112 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:49.112 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:48:02.212603Z" 2020-11-09T12:48:49.113 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:49.113 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:49.113 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:48:49.113 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:49.113 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:48:49.113 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 0, 2020-11-09T12:48:49.113 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 0 2020-11-09T12:48:49.114 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:49.114 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:48:49.115 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-09T12:48:49.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:49.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:49.116 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:49.153 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:49.153 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:49.393 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:48:49.394 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:49.394 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-09T12:48:49.394 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:46:56.199479Z" 2020-11-09T12:48:49.394 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:49.395 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:49.395 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T12:48:49.395 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:48:49.395 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:49.395 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:49.396 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-09T12:48:49.396 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:46:09.701104Z" 2020-11-09T12:48:49.396 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:49.396 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:49.396 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-09T12:48:49.397 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:48:02.212603Z" 2020-11-09T12:48:49.397 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:49.397 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:48:49.398 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-09T12:48:49.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:49.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:49.399 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:49.442 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:49.442 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:49.710 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:48:49.710 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:48:49.710 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:48:49.710 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:48:49.711 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:48:49.711 INFO:teuthology.orchestra.run.smithi190.stdout: "instance_id": 12063671018344825258 2020-11-09T12:48:49.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:49.711 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:48:49.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:49.711 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:48:49.712 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:49.712 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "incremental-sync", 2020-11-09T12:48:49.712 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-09T12:48:49.712 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:49.712 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:49.712 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:49.712 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:46:56.199479Z" 2020-11-09T12:48:49.712 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:49.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:49.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:49.713 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:48:49.713 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:49.713 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "incremental-sync", 2020-11-09T12:48:49.713 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T12:48:49.713 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:49.714 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:49.714 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:49.714 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000" 2020-11-09T12:48:49.714 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:49.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:49.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:49.715 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:48:49.715 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:49.715 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "incremental-sync", 2020-11-09T12:48:49.715 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-09T12:48:49.715 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:49.716 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:49.716 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:49.716 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:46:09.701104Z" 2020-11-09T12:48:49.716 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:49.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:49.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:49.716 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:48:49.717 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:49.717 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "incremental-sync", 2020-11-09T12:48:49.717 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-09T12:48:49.717 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:49.717 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:49.717 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:49.717 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:48:02.212603Z" 2020-11-09T12:48:49.718 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:49.718 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:49.718 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:48:49.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:49.718 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:48:49.718 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 0, 2020-11-09T12:48:49.718 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 0 2020-11-09T12:48:49.718 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:49.719 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:48:49.720 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-09T12:48:49.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:49.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:49.721 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:49.804 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:49.805 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:50.060 INFO:teuthology.orchestra.run.smithi190.stdout:[ 2020-11-09T12:48:50.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:50.061 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-11-09T12:48:50.061 INFO:teuthology.orchestra.run.smithi190.stdout: "last_update": "2020-11-09T12:46:56.367043Z" 2020-11-09T12:48:50.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:50.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:50.061 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T12:48:50.061 INFO:teuthology.orchestra.run.smithi190.stdout: "last_update": "0.000000" 2020-11-09T12:48:50.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:50.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:50.062 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-11-09T12:48:50.062 INFO:teuthology.orchestra.run.smithi190.stdout: "last_update": "2020-11-09T12:46:09.606778Z" 2020-11-09T12:48:50.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:50.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:50.063 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-09T12:48:50.063 INFO:teuthology.orchestra.run.smithi190.stdout: "last_update": "2020-11-09T12:48:02.380948Z" 2020-11-09T12:48:50.063 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:50.063 INFO:teuthology.orchestra.run.smithi190.stdout:] 2020-11-09T12:48:50.064 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-09T12:48:50.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:50.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:50.065 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:50.110 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:50.110 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:50.366 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:48:50.367 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:48:50.367 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:48:50.367 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:48:50.367 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:48:50.367 INFO:teuthology.orchestra.run.smithi164.stdout: "instance_id": 13534362518521853732 2020-11-09T12:48:50.367 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:50.368 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:48:50.368 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:50.368 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:48:50.368 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:50.368 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:50.368 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-11-09T12:48:50.368 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:50.369 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:50.369 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:50.369 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:46:56.367043Z" 2020-11-09T12:48:50.369 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:50.369 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:50.369 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:50.369 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:48:50.370 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:50.370 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:50.370 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T12:48:50.370 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:50.370 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:50.370 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:50.370 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000" 2020-11-09T12:48:50.371 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:50.371 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:50.371 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:50.371 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:48:50.371 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:50.371 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:50.371 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-11-09T12:48:50.372 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:50.372 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:50.372 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:50.372 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:46:09.606778Z" 2020-11-09T12:48:50.372 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:50.372 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:50.372 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:50.372 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:48:50.373 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:50.373 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:50.373 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-09T12:48:50.373 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:50.373 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:50.373 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:50.373 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:48:02.380948Z" 2020-11-09T12:48:50.374 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:50.374 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:50.374 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:48:50.375 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:50.375 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:48:50.375 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 0, 2020-11-09T12:48:50.375 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 0 2020-11-09T12:48:50.375 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:50.376 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:48:50.377 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-09T12:48:50.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:50.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:50.378 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:50.418 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:50.419 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:50.659 INFO:teuthology.orchestra.run.smithi190.stdout:[ 2020-11-09T12:48:50.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:50.659 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-11-09T12:48:50.659 INFO:teuthology.orchestra.run.smithi190.stdout: "last_update": "2020-11-09T12:46:56.367043Z" 2020-11-09T12:48:50.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:50.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:50.660 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T12:48:50.660 INFO:teuthology.orchestra.run.smithi190.stdout: "last_update": "0.000000" 2020-11-09T12:48:50.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:50.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:50.661 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-11-09T12:48:50.661 INFO:teuthology.orchestra.run.smithi190.stdout: "last_update": "2020-11-09T12:46:09.606778Z" 2020-11-09T12:48:50.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:50.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:50.661 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-09T12:48:50.661 INFO:teuthology.orchestra.run.smithi190.stdout: "last_update": "2020-11-09T12:48:02.380948Z" 2020-11-09T12:48:50.662 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:50.662 INFO:teuthology.orchestra.run.smithi190.stdout:] 2020-11-09T12:48:50.663 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-09T12:48:50.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:50.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:50.665 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:50.706 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:50.707 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:50.951 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:48:50.951 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:48:50.951 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:48:50.951 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:48:50.952 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:48:50.952 INFO:teuthology.orchestra.run.smithi164.stdout: "instance_id": 18379200754165384890 2020-11-09T12:48:50.952 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:50.952 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:48:50.952 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:50.952 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:48:50.953 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:50.953 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:50.953 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-11-09T12:48:50.953 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:50.953 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:50.953 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:50.953 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:46:56.367043Z" 2020-11-09T12:48:50.954 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:50.954 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:50.954 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:50.954 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:48:50.954 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:50.954 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:50.955 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T12:48:50.955 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:50.955 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:50.955 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:50.955 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000" 2020-11-09T12:48:50.955 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:50.955 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:50.956 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:50.956 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:48:50.956 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:50.956 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:50.956 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-11-09T12:48:50.956 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:50.956 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:50.957 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:50.957 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:46:09.606778Z" 2020-11-09T12:48:50.957 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:50.957 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:50.957 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:50.957 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:48:50.958 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:50.958 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:50.958 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-09T12:48:50.958 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:50.958 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:50.958 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:50.958 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:48:02.380948Z" 2020-11-09T12:48:50.959 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:50.959 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:50.959 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:48:50.959 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:50.959 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:48:50.959 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 0, 2020-11-09T12:48:50.959 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 0 2020-11-09T12:48:50.960 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:50.960 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:48:50.961 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-09T12:48:50.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:50.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:50.962 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:51.044 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:51.044 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:51.280 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:48:51.281 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:51.281 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000005167", 2020-11-09T12:48:51.281 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:46:56.195391Z" 2020-11-09T12:48:51.281 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:51.281 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:51.281 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T12:48:51.282 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:48:51.282 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:51.282 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:51.282 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-09T12:48:51.282 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:46:09.705425Z" 2020-11-09T12:48:51.282 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:51.283 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:51.283 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-09T12:48:51.283 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:47:40.204906Z" 2020-11-09T12:48:51.283 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:51.283 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:48:51.284 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-09T12:48:51.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:51.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:51.285 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:51.368 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:51.369 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:51.625 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:48:51.626 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:48:51.626 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:48:51.626 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:48:51.626 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:48:51.626 INFO:teuthology.orchestra.run.smithi164.stdout: "instance_id": 13365629761463538848 2020-11-09T12:48:51.626 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:51.626 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:48:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:48:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000005167", 2020-11-09T12:48:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:51.628 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:51.628 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:46:56.195391Z" 2020-11-09T12:48:51.628 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:51.628 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:51.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:51.628 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:48:51.629 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:51.629 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:51.629 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T12:48:51.629 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:51.629 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:51.629 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:51.629 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000" 2020-11-09T12:48:51.630 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:51.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:51.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:51.630 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:48:51.630 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:51.630 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:51.630 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-09T12:48:51.631 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:51.631 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:51.631 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:51.631 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:46:09.705425Z" 2020-11-09T12:48:51.631 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:51.631 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:51.631 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:51.631 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:48:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "incremental-sync", 2020-11-09T12:48:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-09T12:48:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:47:40.204906Z" 2020-11-09T12:48:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:48:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:48:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 0, 2020-11-09T12:48:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 0 2020-11-09T12:48:51.634 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:51.634 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:48:51.635 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-09T12:48:51.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:51.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:51.636 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:51.724 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:51.724 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:51.963 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:48:51.963 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:51.964 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000005167", 2020-11-09T12:48:51.964 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:46:56.195391Z" 2020-11-09T12:48:51.964 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:51.964 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:51.964 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T12:48:51.964 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:48:51.964 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:51.965 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:51.965 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-09T12:48:51.965 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:46:09.705425Z" 2020-11-09T12:48:51.965 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:51.965 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:51.965 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-09T12:48:51.965 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:47:40.204906Z" 2020-11-09T12:48:51.966 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:51.966 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:48:51.967 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-09T12:48:51.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:51.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:51.968 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:52.012 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:52.013 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:52.271 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:48:52.271 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:48:52.271 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:48:52.271 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:48:52.272 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:48:52.272 INFO:teuthology.orchestra.run.smithi190.stdout: "instance_id": 4717572788910005996 2020-11-09T12:48:52.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:52.272 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:48:52.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:52.272 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:48:52.273 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:52.273 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "incremental-sync", 2020-11-09T12:48:52.273 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000005167", 2020-11-09T12:48:52.273 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:52.273 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:52.273 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:52.273 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:46:56.195391Z" 2020-11-09T12:48:52.273 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:52.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:52.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:52.274 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:48:52.274 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:52.274 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "incremental-sync", 2020-11-09T12:48:52.274 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-09T12:48:52.275 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:52.275 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:52.275 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:52.275 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000" 2020-11-09T12:48:52.275 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:52.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:52.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:52.275 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:48:52.276 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:52.276 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "incremental-sync", 2020-11-09T12:48:52.276 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-09T12:48:52.276 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:52.276 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:52.276 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:52.276 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:46:09.705425Z" 2020-11-09T12:48:52.277 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:52.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:52.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:52.277 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:48:52.277 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:52.277 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "incremental-sync", 2020-11-09T12:48:52.277 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-09T12:48:52.277 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:52.278 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:52.278 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:52.278 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:47:40.204906Z" 2020-11-09T12:48:52.278 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:52.278 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:52.278 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:48:52.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:52.279 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:48:52.279 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 0, 2020-11-09T12:48:52.279 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 0 2020-11-09T12:48:52.279 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:52.279 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:48:52.280 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-09T12:48:52.281 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2020-11-09T12:48:52.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:52.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:52.666 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:52.702 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:52.702 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:52.916 INFO:teuthology.orchestra.run.smithi164.stderr:No --period given, using current period=d8c95da7-75a8-4dcf-94ba-f12d5c80a46f 2020-11-09T12:48:52.946 INFO:teuthology.orchestra.run.smithi164.stdout:[ 2020-11-09T12:48:52.946 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:52.946 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604926132.660709_98.1", 2020-11-09T12:48:52.946 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:48:52.660709Z" 2020-11-09T12:48:52.946 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:52.947 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:52.947 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:48:52.947 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "2020-11-09T12:42:16.653632Z" 2020-11-09T12:48:52.947 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:52.947 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:52.947 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:48:52.947 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:48:52.948 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:52.948 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:52.948 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:48:52.948 INFO:teuthology.orchestra.run.smithi164.stdout: "last_update": "0.000000" 2020-11-09T12:48:52.948 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:52.949 INFO:teuthology.orchestra.run.smithi164.stdout:] 2020-11-09T12:48:52.950 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-09T12:48:52.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:52.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:52.951 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:52.993 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:52.994 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:53.255 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:48:53.255 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:48:53.256 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:48:53.256 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:48:53.256 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:48:53.256 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:48:53.256 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:48:53.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:53.257 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:48:53.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:53.257 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:48:53.257 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:53.257 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:48:53.257 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604926132.660709_98.1", 2020-11-09T12:48:53.257 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:53.258 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:48:53.258 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:53.258 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:48:52.660709Z", 2020-11-09T12:48:53.258 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:48:53.258 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:53.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:53.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:53.259 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:48:53.259 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:53.259 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:48:53.259 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:48:53.260 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:53.260 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:53.260 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:53.260 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:48:53.260 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:48:53.260 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:53.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:53.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:53.261 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:48:53.261 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:53.261 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:48:53.261 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:48:53.261 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:53.261 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:53.261 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:53.262 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:48:53.262 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:48:53.262 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:53.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:53.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:53.262 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:48:53.262 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:53.263 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:48:53.263 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:48:53.263 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:53.263 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:53.263 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:53.263 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:48:53.263 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:48:53.264 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:53.264 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:53.264 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:48:53.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:53.264 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:48:53.264 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:48:53.264 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:48:53.264 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:53.265 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:48:53.266 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-09T12:48:53.266 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-09T12:48:53.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:53.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:53.267 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:53.308 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:53.309 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:53.571 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-11-09T12:48:53.571 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": { 2020-11-09T12:48:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: "info": { 2020-11-09T12:48:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: "status": "sync", 2020-11-09T12:48:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: "num_shards": 4, 2020-11-09T12:48:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:48:53.573 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:48:53.573 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:53.573 INFO:teuthology.orchestra.run.smithi164.stdout: "markers": [ 2020-11-09T12:48:53.573 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:53.574 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 0, 2020-11-09T12:48:53.574 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:53.574 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:48:53.574 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604926132.660709_98.1", 2020-11-09T12:48:53.574 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:53.575 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 1, 2020-11-09T12:48:53.575 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:53.575 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:48:52.660709Z", 2020-11-09T12:48:53.575 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:48:53.575 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:53.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:53.575 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:53.575 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 1, 2020-11-09T12:48:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:48:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:48:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:48:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:48:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 2, 2020-11-09T12:48:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:48:53.579 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:48:53.579 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:53.579 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:53.579 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:53.579 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:48:53.579 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:48:53.579 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:53.580 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:53.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-11-09T12:48:53.580 INFO:teuthology.orchestra.run.smithi164.stdout: "key": 3, 2020-11-09T12:48:53.580 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2020-11-09T12:48:53.580 INFO:teuthology.orchestra.run.smithi164.stdout: "state": 1, 2020-11-09T12:48:53.580 INFO:teuthology.orchestra.run.smithi164.stdout: "marker": "", 2020-11-09T12:48:53.580 INFO:teuthology.orchestra.run.smithi164.stdout: "next_step_marker": "", 2020-11-09T12:48:53.581 INFO:teuthology.orchestra.run.smithi164.stdout: "total_entries": 0, 2020-11-09T12:48:53.581 INFO:teuthology.orchestra.run.smithi164.stdout: "pos": 0, 2020-11-09T12:48:53.581 INFO:teuthology.orchestra.run.smithi164.stdout: "timestamp": "0.000000", 2020-11-09T12:48:53.581 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2020-11-09T12:48:53.581 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:53.581 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:53.581 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-11-09T12:48:53.581 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-11-09T12:48:53.582 INFO:teuthology.orchestra.run.smithi164.stdout: "full_sync": { 2020-11-09T12:48:53.582 INFO:teuthology.orchestra.run.smithi164.stdout: "total": 1, 2020-11-09T12:48:53.582 INFO:teuthology.orchestra.run.smithi164.stdout: "complete": 1 2020-11-09T12:48:53.582 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-11-09T12:48:53.582 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-11-09T12:48:53.583 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-09T12:48:53.584 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-09T12:48:53.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:53.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-09T12:48:53.586 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-09T12:48:53.626 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:53.627 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:53.895 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2020-11-09T12:48:53.896 INFO:teuthology.orchestra.run.smithi190.stdout: "sync_status": { 2020-11-09T12:48:53.896 INFO:teuthology.orchestra.run.smithi190.stdout: "info": { 2020-11-09T12:48:53.896 INFO:teuthology.orchestra.run.smithi190.stdout: "status": "sync", 2020-11-09T12:48:53.896 INFO:teuthology.orchestra.run.smithi190.stdout: "num_shards": 4, 2020-11-09T12:48:53.896 INFO:teuthology.orchestra.run.smithi190.stdout: "period": "d8c95da7-75a8-4dcf-94ba-f12d5c80a46f", 2020-11-09T12:48:53.896 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:48:53.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:53.897 INFO:teuthology.orchestra.run.smithi190.stdout: "markers": [ 2020-11-09T12:48:53.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:53.897 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 0, 2020-11-09T12:48:53.897 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:53.897 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:48:53.897 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604926132.660709_98.1", 2020-11-09T12:48:53.897 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:53.898 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 1, 2020-11-09T12:48:53.898 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:53.898 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:48:52.660709Z", 2020-11-09T12:48:53.898 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:48:53.898 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:53.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:53.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:53.899 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 1, 2020-11-09T12:48:53.899 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:53.899 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:48:53.899 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "1_1604925736.653632_120.1", 2020-11-09T12:48:53.899 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:53.900 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:53.900 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:53.900 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "2020-11-09T12:42:16.653632Z", 2020-11-09T12:48:53.900 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:48:53.900 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:53.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:53.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:53.901 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 2, 2020-11-09T12:48:53.901 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:53.901 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:48:53.901 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:48:53.901 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:53.901 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:53.901 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:53.902 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:48:53.902 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:48:53.902 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:53.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:53.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-11-09T12:48:53.903 INFO:teuthology.orchestra.run.smithi190.stdout: "key": 3, 2020-11-09T12:48:53.903 INFO:teuthology.orchestra.run.smithi190.stdout: "val": { 2020-11-09T12:48:53.903 INFO:teuthology.orchestra.run.smithi190.stdout: "state": 1, 2020-11-09T12:48:53.903 INFO:teuthology.orchestra.run.smithi190.stdout: "marker": "", 2020-11-09T12:48:53.904 INFO:teuthology.orchestra.run.smithi190.stdout: "next_step_marker": "", 2020-11-09T12:48:53.904 INFO:teuthology.orchestra.run.smithi190.stdout: "total_entries": 0, 2020-11-09T12:48:53.905 INFO:teuthology.orchestra.run.smithi190.stdout: "pos": 0, 2020-11-09T12:48:53.905 INFO:teuthology.orchestra.run.smithi190.stdout: "timestamp": "0.000000", 2020-11-09T12:48:53.905 INFO:teuthology.orchestra.run.smithi190.stdout: "realm_epoch": 2 2020-11-09T12:48:53.905 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:53.905 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:53.905 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2020-11-09T12:48:53.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-11-09T12:48:53.906 INFO:teuthology.orchestra.run.smithi190.stdout: "full_sync": { 2020-11-09T12:48:53.906 INFO:teuthology.orchestra.run.smithi190.stdout: "total": 1, 2020-11-09T12:48:53.906 INFO:teuthology.orchestra.run.smithi190.stdout: "complete": 1 2020-11-09T12:48:53.906 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-11-09T12:48:53.906 INFO:teuthology.orchestra.run.smithi190.stdout:} 2020-11-09T12:48:53.907 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-09T12:48:54.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:48:54.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:48:54.812 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:48:54.860 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:48:54.860 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:48:55.082 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:48:55.078+0000 7f881471d0c0 1 waiting for incremental sync to catch up: 2020-11-09T12:48:55.082 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , , , , , , ] 2020-11-09T12:48:55.082 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , , , , 00000000001.47.5, , ] 2020-11-09T12:49:00.085 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:49:00.082+0000 7f881471d0c0 1 bucket sync caught up with source: 2020-11-09T12:49:00.085 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , , , , 00000000001.47.5, , ] 2020-11-09T12:49:00.085 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , , , , 00000000001.47.5, , ] 2020-11-09T12:49:00.086 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:49:00.082+0000 7f881471d0c0 0 bucket checkpoint complete 2020-11-09T12:49:00.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:49:00.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:49:00.116 INFO:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kuljus-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-09T12:49:00.157 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:49:00.158 INFO:teuthology.orchestra.run.smithi164.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:49:00.392 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:49:00.389+0000 7fd0e82c70c0 1 bucket sync caught up with source: 2020-11-09T12:49:00.392 INFO:teuthology.orchestra.run.smithi164.stderr: local status: [, , , , , , , , 00000000001.47.5, , ] 2020-11-09T12:49:00.392 INFO:teuthology.orchestra.run.smithi164.stderr: remote markers: [, , , , , , , , 00000000001.47.5, , ] 2020-11-09T12:49:00.393 INFO:teuthology.orchestra.run.smithi164.stderr:2020-11-09T12:49:00.389+0000 7fd0e82c70c0 0 bucket checkpoint complete 2020-11-09T12:49:00.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:49:00.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kuljus-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-09T12:49:00.427 INFO:teuthology.orchestra.run.smithi190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kuljus-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-09T12:49:00.475 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setuser ceph since I am not root 2020-11-09T12:49:00.475 INFO:teuthology.orchestra.run.smithi190.stderr:ignoring --setgroup ceph since I am not root 2020-11-09T12:49:00.722 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:49:00.718+0000 7f88cafaf0c0 1 bucket sync caught up with source: 2020-11-09T12:49:00.723 INFO:teuthology.orchestra.run.smithi190.stderr: local status: [, , , , , , , , 00000000001.47.5, , ] 2020-11-09T12:49:00.723 INFO:teuthology.orchestra.run.smithi190.stderr: remote markers: [, , , , , , , , 00000000001.47.5, , ] 2020-11-09T12:49:00.723 INFO:teuthology.orchestra.run.smithi190.stderr:2020-11-09T12:49:00.718+0000 7f88cafaf0c0 0 bucket checkpoint complete 2020-11-10T00:20:31.923 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-11-10T00:20:31.956 DEBUG:teuthology.task.console_log:Killing console logger for smithi164 2020-11-10T00:20:31.957 DEBUG:teuthology.task.console_log:Killing console logger for smithi190 2020-11-10T00:20:31.958 DEBUG:teuthology.task.console_log:Killing console logger for smithi164 2020-11-10T00:20:31.958 DEBUG:teuthology.task.console_log:Killing console logger for smithi190 2020-11-10T00:20:31.958 DEBUG:teuthology.exit:Finished running handlers